Playing with numbers - Scottish Bike Counters

Specific examples and ideas, or generic types.

Moderator: LeeGCU

Forum rules
Don't swear. Be Nice. Superb. More details here.

Playing with numbers - Scottish Bike Counters

Postby freeyourinnertube » 27 Feb 2015 19:18

I have been doing some software development that uses the Scottish counter stats, met data, and Glasgow holiday data (public and school), and I know there is also live traffic data that I could also use (but haven't yet). The end result is an online pivot chart that allows questions such as, how many cyclists are there on a rainy Tuesday in February?; or how many cyclists stop cycling on a Wednesday if the wind speed is > 20 mph?; or how many cyclists are there on a cold Thursday in January, cf a cold Thursday in February? - or a miriad of other questions. The granularity of the data that I've been collecting is 1 hour.

I know that the stats on the Danish website stopped being updated last Friday -I'd be interested in knowing whether the counter on the Anderston bridge is working.

Anyone know?
Last edited by freeyourinnertube on 08 Apr 2015 13:08, edited 1 time in total.
freeyourinnertube
 
Posts: 68
Joined: 26 Nov 2013 23:44

Re: Playing with numbers

Postby iai » 28 Feb 2015 14:13

I'll have a look to see if its working next time I go past it. Middle meadow walk seemed to be working when I saw it last night. (About 1300)
iai
 
Posts: 77
Joined: 26 Nov 2013 12:08

Re: Playing with numbers

Postby freeyourinnertube » 01 Mar 2015 11:01

Thanks.

I suspected the counters were working, just the central graphs not - since posting the counter in Perth (BC4) seems to have started publishing data.
I'm waiting for the OK from Sustrans before making my site available - I checked with them as there are no use or copyright statements on the site that I get the data from. If I'm not allowed to republish I'll make the software available so that individuals can also log and look at the data themselves.

Anyone interested in what I'm on about should look at http://nicolas.kruchten.com/pivottable/ ... index.html which is a demo of the online pivot chart tool that I'm using and you'll begin to see what's possible, only I'm using all Scottish counter data, meterologic data, holiday data, and perhaps local road travel data too.
freeyourinnertube
 
Posts: 68
Joined: 26 Nov 2013 23:44

Re: Playing with numbers

Postby joel_c » 01 Mar 2015 15:32

Are you likely to make it along to Cyclehack this year? Would be good to have someone with your skill-set on board.
joel_c
 
Posts: 427
Joined: 26 Nov 2013 17:02

Re: Playing with numbers

Postby freeyourinnertube » 01 Mar 2015 17:02

I'm not short of stuff to do in my 'spare' time, I'm a bit high on the old duffer scale cf the cyclehack video link you posted, and tbh I'm not 100% convinced by cyclehack, other than the broad enthusiasm shown - we'll see.

pm sent
freeyourinnertube
 
Posts: 68
Joined: 26 Nov 2013 23:44

Re: Playing with numbers

Postby freeyourinnertube » 27 Mar 2015 20:00

Had a quick meeting with an officer from GCC today (very helpful and productive), and I now have the raw data from the Anderston Bridge counter going back to July 2014, and permission is being requested to allow my use of the data on a public website, and to have new count data sent to me. The data is better than that which i've been able to harvest from the Danish counter site, in that the counter is bi-directional - so I'll be able to plot trips in each direction, and it also records the speeds of the cyclists (though since it's just before a turn is likely not to be particularly useful for this counter) and temperature.

I should also be able to identify those in other councils who control similar data, with a view to requesting access to the raw data for those counters now too.

Bit of work to do to import the data into the database but things took a big jump forwards today.

*UPDATE*
Data now in database (about 46000 datapoints) and can now do some basic manipulations - already interesting information re direction of travel that I couldn't look at earlier - need to check the imported data for errors and correct for GMT/BST but enough for today.
freeyourinnertube
 
Posts: 68
Joined: 26 Nov 2013 23:44

Re: Playing with numbers - Scottish Bike Counters

Postby freeyourinnertube » 08 Apr 2015 12:33

Glasgow city council have given me permission to publish the Anderston Bridge stats via my website
now live at

http://178.62.84.220/bikestats/index.html

As above I've imported the official raw data and will be making that live too very soon.

At the moment you can only see one counter site, but I'm collecting data from them all, and will seek permission to publish them too.
There's a big spike in the graph shown on my site, this is not an error, it's caused by the counters stopping updating the web page (but continuing to count), when they restarted, all the interim cyclist counts appeared - this sort of problem will disappear once I get regular access to the full raw data (any day soon).

There's a few weeks worth of data available now, but the 46000 datapoints mentioned above will appear, probably by the weekend. From eyeballing the official raw data there's a directional difference in use of the bridge, though since I've yet to receive official labeling of the directions I can only infer it from the speed data - more when I make that data live.
freeyourinnertube
 
Posts: 68
Joined: 26 Nov 2013 23:44

Re: Playing with numbers - Scottish Bike Counters

Postby freeyourinnertube » 12 Apr 2015 19:02

Both the data I harvest (last month), and the official raw count data July 2014 -> March 2015 for the Anderston Bridge counter is now available via

http://178.62.84.220/bikestats/index.html

There is a directional bias in the journeys across the bridge, but I have to infer the direction from the speed data - getting confirmation about that - have a play with the different graphs, the tidal commute is obvious in some representations.

Will try to add weather data to that shortly

Nextbike site still collecting data though I need to rework the script as it's timing out during data processing, so the stats aren't available at present - basically as the datasets grow I'll have to become more efficient with processing and cacheing.
freeyourinnertube
 
Posts: 68
Joined: 26 Nov 2013 23:44

Re: Playing with numbers - Scottish Bike Counters

Postby iai » 26 Apr 2015 11:02

@ freeyourinnertube.
Sorry I forgot to reply to you about the Anderston Counter. That's a good source of data thanks!
I wonder what happened on the day where 700 counts happened?

Were you intending to include other counters as well?

I'd be up for helping with the project at cycle hack or otherwise!
iai
 
Posts: 77
Joined: 26 Nov 2013 12:08

Re: Playing with numbers - Scottish Bike Counters

Postby freeyourinnertube » 26 Apr 2015 21:05

iai wrote:@ freeyourinnertube.
Sorry I forgot to reply to you about the Anderston Counter. That's a good source of data thanks!
I wonder what happened on the day where 700 counts happened?

Were you intending to include other counters as well?

I'd be up for helping with the project at cycle hack or otherwise!


Hi iai

It's getting quite complicated, but I'll quickly go over the current status.

I have access from GCC to the raw counter stats (historic)
I am now emailed daily official raw stats for the Anderston counter
I have permission to make these stats public on my site (which I'm doing)

I currently harvest data for all the Scottish counters, Dublin, and Copenhagen, each hour, as well as the weather data for each of these sites - I've got code that displays them, I've just not made them public - I've been doing this for 2 months.
I am going to contact the councils for each of the other counters, in order to get permission to publish their data too. I've been in contact with Zenbike UK as he's doing something similar (http://www.zenbike.co.uk/cycling/bike_c ... index.html), and I've shared some code with him as he was grabbing the data in a different way to me, and we've exchanged information as we've discovered who to contact re the data (the hard part, not the coding). We've each had meetings with some councils/stakeholders. The GCC contact I have has been very helpful and this will allow my to contact the correct people re permission.

I need to write a little script that downloads the csv raw data email attachments and process them into the database (will be easy, just not got around to it yet)
I need to get hold of the raw stats for the the few days that passed between my bulk insertion of raw data, and the date on which I started getting emailed statistics.

As it turns out, I may be doing some paid work on other Glasgow datasets, possibly based on the work I have done already.
I'm not sure whether I'll be going to cyclehack (probably not), and not sure how the paid work will impact any offers of help (thanks by the way).


The 700 figure is a mystery to me, I've noticed a bug in the graphs on the Danish site (spike at midnight) and reported it to the company. There are other spikes that I know the cause of though. The counters (or some software - details sketchy) has been bothered by bots, so the Scottish counters were taken offline (but were still counting) until some counter-measures (I'm aware of the pun) were added to the software. When they came back, my graphs spike as I plot the graphs based on DeltaYearcount, since they were off for a number of weeks, my counts therefore jump by all the counted, but not communicated, total. There are also sometimes funny values coming back from the counters, and I'm not sure what causes that, I added some logic to try and filter these spikes out - this isn't an issue with the official data, just my web-harvested data.

Hope this is of interest - one thing I instantly noticed is the bias in direction of travel across the bridge - anyone got any ideas why this may be so (4%ish difference after 35000 trips), difference is visible most days?
freeyourinnertube
 
Posts: 68
Joined: 26 Nov 2013 23:44

Next

Return to Infrastructure stuff

Who is online

Users browsing this forum: No registered users and 1 guest