Using cURL to Read the Contents of a Web Page

Categories

Web Design

Recently I wrote about how to use the Yahoo! weather api with WordPress and in the comments I was asked how to use it without relying on WordPress. The answer – is cURL.

In this post I’m going to show you how to use cURL with PHP to read the contents of a front-end api, but it could just easily be used for scraping a web page, or accessing a REST api.

I’m going to assume that we are still using PHP since some programming is still needed. However you can also use cURL with other languages such as Ruby, Python, Perl, and also on the command line.

According to Wikipedia the name cURL comes from “Client for URLs” and it is essentially a command line interface for a web client. This means that you can access web content through a script on your site. This is most often used by websites when they access web apis such as Twitter, Flickr, or as in this case, the Yahoo! weather api.

Note: There’s actually loads of different commands and settings for cURL but we are only interested in a few. If you want to check them all out then you can view the docs on php.net.

Below is the code we will be using:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $file);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $_SERVER['REQUEST_URI']);
$result = curl_exec($ch);
curl_close($ch);

Broken down we have:

  • $ch = curl_init(); intiate the curl object
  • curl_setopt($ch, CURLOPT_URL, $file); specify the file or url to load
  • curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); tell it to return the raw content
  • curl_setopt($ch, CURLOPT_REFERER, $_SERVER[‘REQUEST_URI’]); Simulate the http referer
  • $result = curl_exec($ch); perform the cURL request
  • curl_close($ch); close the connection

A bit of rejigging from the original WordPress Yahoo! post and you end up with:

<?php
function bm_getWeather ($code = '', $temp = 'c') {
	$file = 'http://weather.yahooapis.com/forecastrss?p=' . $code . '&u=' . $temp;

	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, $file);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	curl_setopt($ch, CURLOPT_REFERER, $_SERVER['REQUEST_URI']);
	$result = curl_exec($ch);
	curl_close($ch);

	$output = array (
		'temperature' => bm_getWeatherProperties('temp', $result),
		'weather' => bm_getWeatherProperties('text', $result),
		'weather_code' => bm_getWeatherProperties('code', $result),
		'class' => 'weatherIcon-' . bm_getWeatherProperties('code', $result),
	);

	return $output;
}

function bm_getWeatherProperties ($needle, $data) {
	$regex = '<yweather:condition.*' . $needle . '="(.*?)".*/>';
	preg_match($regex, $data, $matches);
	return $matches[1];
}

Note: The weather API I used here is no longer available but the cURL code is still perfectly valid.

Let me know what you think on Mastodon, or BlueSky (or Twitter X if you must).

Related Posts

07 Mar 2010

WordPress Http API – Read Content From Other Websites

Recently I have been doing a lot of work with API’s and these often involve loading text content (rss/ xml/ json etc) from another website and then displaying the results – for instance the Twitter feed at the bottom of...
30 Jun 2007

WordPress tips and tricks – Custom Page Templates

Some time ago I posted the first of my tips and tricks for WordPress, and I thought it was about time I posted some more so, to start things off, here is a short tutorial on custom page templates in...
28 Aug 2008

WordPress 2.7 and Crazyhorse

Last weekend I spent a few days away from WordPress and work in general to spend some quality time with the girlfriend. When I got back and updated my local copy of WordPress (via svn) I was surprised to see...
10 Mar 2010

Using the Yahoo Weather API (in your WordPress themes)

When developing our Hyperlocal WordPress theme called The Local Darren came up with the idea of adding the local weather to the themes header. After a bit of research I decided that the Yahoo! Weather api was the best solution.Before...
29 Jul 2010

Easy WordPress Updates: Store FTP Info in wp-config.php

Saw an interesting blog post on Twitter today about storing WordPress FTP information in wp-config.php. The article was written in German so I sent the author an email to ask if he’d mind me translating it. Phil, the author, very...