Downloading Web Content and Files :: Data Import and Export (Programming)

Programming

Downloading Web Content and Files

MATLAB provides two functions for downloading Web pages and files using HTTP: urlread and urlwrite. With the urlread function, you can read and save the contents of a Web page to a string variable in the MATLAB workspace. With the urlwrite function, you can save a Web page's content to a file.

Because it creates a string variable in the workspace, the urlread function is useful for working with the contents of Web pages in MATLAB. The urlwrite function is useful for saving Web pages to a local directory.

Note When using urlread, remember that only the HTML in that specific Web page is retrieved. The hyperlink targets, images, and so on will not be retrieved.

If you need to pass parameters to a Web page, the urlread and urlwrite functions let you use HTTP post and get methods. For more information, see the urlread and urlwrite reference pages.

Example -- Using the urlread Function

The following procedure demonstrates how to retrieve the contents of the Web page containing the Recent File list at the MATLAB Central File Exchange, http://www.mathworks.com/matlabcentral/fileexchange/index.jsp. It assigns the results to a string variable, newFiles, and it uses the strfind function to search the retrieved content for a specific word:

Retrieve the Web page content with the urlread function:

recentFile = 
urlread('http://www.mathworks.com/matlabcentral/fileexchange/lo
adFileList.do?objectType=fileexchange&orderBy=date&srt3=0');

After retrieving the content, run the strfind function on the recentFile variable:

```
hits = findstr(recentFile,'Simulink');
```

If the file contains the word Simulink, MATLAB will store the matches in the hits variable.

While you can manually pass arguments using the URL, the urlread function also lets you pass parameters to a Web page using standard HTTP methods, including post and form. Using the HTTP get method, which passes parameters in the URL, the following code queries Google for the word Simulink:

s = 
urlread('http://www.google.com/search','get',{'q','Simulink'})

For more information, see the urlread reference page.

Example -- Using the urlwrite Function

The following example builds on the procedure in the previous section. This example still uses urlread and checks for a specific word, but it also uses urlwrite to save the file if it contains any matches:

%The urlread function loads the contents of the Web page into the 
%MATLAB workspace.
recentFile = 
urlread('http://www.mathworks.com/matlabcentral/fileexchange/lo
adFileList.do?objectType=fileexchange&orderBy=date&srt3=0');

%The findstr function searches for the word "Simulink".
hits = findstr(recentFile,'Simulink');

%The if statement checks for any hits. 
if(isempty(hits) == 0)
    %If there are hits, the Web page will be saved locally using 
the
   %urlwrite function.
    
urlwrite('http://www.mathworks.com/matlabcentral/fileexchange/l
oadFileList.do?objectType=fileexchange&orderBy=date&srt3=0','co
ntains_simulink.html');
end;

Note that the Web page is saved as contains_simulink.html.

Exchanging Files over the Internet Creating and Uncompressing Zip Archives