From hk0weather to Open Data, Open Government

從 hk0weather 到 Open Data, Open Government

From hk0weather to Open Data, Open
Government

Sammy Fung
BarCampHK 2013

http://slidesha.re/11ZPKvx

Sammy Fung
● Blogger: sammy.hk
● Coder: perl -> php -> python
● Linux User: server + desktop
● “Cloud”: OpenStack.
● Smartphone: iPhone 3G -> Android
● Organizer: Linux/Open Source Communities

Agenda
● My Products: WeatherHK, TCTrack
● My new opensource projects: hk0weather
● My comments to Open Data and Open
Government
● Q&A, Open Discussion to Floor.

My Products
● WeatherHK
● TCTrack

WeatherHK
● http://twitter.com/weatherhk
● Interview by MetroPop in 2009.

WeatherHK
● http://twitter.com/weatherhk
● Backend: Python + Scrapy + Database +
Twitter + NNTP......
● Frontend: Twitter + Newsgroup
● hourly current weather report
● weather forecast report
● tropical signal warning

TCTrack
● http://sammy.hk/projects/tctrack/tctrack.php
● http://twitter.com/tctrack
● Plot TC current and forecast tracks over
Google Map.
● Tweet JTWC updates for Northwest Pacific.
● Source: JTWC, HKO.

TCTrack
● http://sammy.hk/projects/tctrack/tctrack.php
● http://twitter.com/tctrack
● Probably first tctrack map in HK using
GoogleMap
● Use of GMap: TCTrack -> Weather
Underground Hong Kong -> HKO

g0v.tw
● g0v.tw 台灣零時政府
● clkao 高嘉良 suggested to implement open
source weather tools for Hong Kong.

hk0weather
● https://github.com/sammyfung/hk0weather
● Open Source Hong Kong Weather Project.
● convert to JSON data from HKO webpages.
● python + scrapy
● 1st version: from current weather report,
extracting temperture and humidity from 20+
weather stations, export in json format.

hk0weather
● https://github.com/sammyfung/hk0weather
● $ virtualenv hk0weatherenv
● $ source hk0weatherenv/bin/activate
● $ pip install scrapy
● $ git clone
https://github.com/sammyfung/hk0weather.git
● $ cd hk0weather
● $ scrapy crawl currwx -t json -o testresult

hk0weather
[{"humidity": 80, "station": "hko", "temperture": 17, "time": 1360785720},
{"station": "kingspark", "temperture": 16, "time": 1360785720},
{"station": "wongchukhang", "temperture": 17, "time": 1360785720},
{"station": "takwuling", "temperture": 16, "time": 1360785720},
{"station": "laufaushan", "temperture": 15, "time": 1360785720},
{"station": "taipo", "temperture": 16, "time": 1360785720},
{"station": "shatin", "temperture": 17, "time": 1360785720},
{"station": "tuenmun", "temperture": 17, "time": 1360785720},
{"station": "tseungkwano", "temperture": 16, "time": 1360785720},
{"station": "saikung", "temperture": 16, "time": 1360785720},
{"station": "cheungchau", "temperture": 17, "time": 1360785720},
{"station": "cheungchau", "temperture": 17, "time": 1360785720},
{"station": "tsingyi", "temperture": 17, "time": 1360785720},
{"station": "shekkong", "temperture": 15, "time": 1360785720},
{"station": "tsuenwanhokoon", "temperture": 15, "time": 1360785720},
{"station": "tsuenwanshingmunvalley", "temperture": 17, "time": 1360785720},
{"station": "hongkongpark", "temperture": 17, "time": 1360785720},
{"station": "shaukeiwan", "temperture": 16, "time": 1360785720},
{"station": "kowlooncity", "temperture": 16, "time": 1360785720},
{"station": "happyvalley", "temperture": 18, "time": 1360785720},
{"station": "wongtaisin", "temperture": 17, "time": 1360785720},
{"station": "stanley", "temperture": 16, "time": 1360785720},
{"station": "kwuntong", "temperture": 15, "time": 1360785720},
{"station": "shamshuipo", "temperture": 17, "time": 1360785720}]

hk0weather
● Future Planning:
● Add more weather reports.
● Getting ideas and/or cooperate with 'pro'
Weather hobbists.
● Remarks:
● Development of hk0weather is started from
ZERO, its code is different than my twitter
@weatherhk.

Why python + scrapy ?
● python: my current favourite programming
language for few years.
● scrapy: web crawling framework written in
Python.

Scrapy
● scrapy: web crawling framework written in
Python.
● HtmlXPathSelector
● Output: built-in JSON, CSV, XML.
● Python: import re

Items.py
class Hk0WeatherItem(Item):
time = Field()
station = Field()
temperture = Field()
humidity = Field()
pass

Currwx.py
start_urls = (

'http://www.weather.gov.hk/wxinfo/currwx/curr
entc.htm',
)

Currwx.py
def parse(self, response):
laststation = ''
temperture = int()
stations = []
hxs = HtmlXPathSelector(response)
report = hxs.select('//div[@id="ming"]')

libhk0
class hk0:
stations = [
(u' 天文台 ', 'hko'),
(u' 京士柏 ', 'kingspark'),
(u' 黃竹坑 ', 'wongchukhang'),
(u' 打鼓嶺 ', 'takwuling'),
(u' 流浮山 ', 'laufaushan'),

libhk0
class hk0:
def gettime(self, report):
…
def hk0current(self, report):
…

Challenge
● Challenge on first day of hk0weather release.
● Facebook comment to blog post: 「其實 HKO
有個年費 plan 係嘔兩個好靚仔嘅 XML 。而只
要你肯問，你件嘢又非商業用途， HKO 都會免
費俾你用」

Challenge
●
I replied after googling for it.
●
google 找到相關 XML schema 示範資料，但 ....
●
1. 在於 Open Data 角度，暫時看不到有網頁說明任何人以非商
業用途使用資訊，可以免費獲得登入戶口。而我本身是「已登
記」使用 HKO 網頁資料作非商業用途的使用 / 轉載者，我一直收
到來自 HKO 的更新電郵都是說明 HTML 格式的更新， " 印象中 "
HKO 並沒有發出過 XML 和邀請開 XML 使用戶口。
●
2. 而 ( 以我覺得 ) 以業餘氣象發燒友的角度來看，目前 HKO XML
schema 仍有不足之處，我已想到一些我想要的數據，仍是只能
從 HTML 版本取得。
●
Challenge: FAIL! XD

Open Data from Govenment
● My Chinese Blog Post 「香港政府機構開放資料
Open Data 情況」 on 2013/1/17.
● Data.One released on 2011/3/31.
● Weather at Data.One provides 7 dataset URLs,
returns RSS (XML) format (Eng/TChi/SChi)
● One word: Useless.
● Data.One dataset is different with HKO own
XML metioned in previous “challenge”.

● Example - Current local weather report:
● Plain text report in RSS.
● Difference to quote report content:
– Website: HTML tags
– Data.One: RSS description tags
● Other weather data is missing, eg. Area
temperture updates per each 12 mins.

● Weather at Data.One is 'report' but not 'data'.
● Weather RSS is already released by HKO
before launch of Data.One.
● Technically, json/xml format is better
readable by computer programs.

Improving Legco Vote Result Data ?
● Legcovotes.net is created by Hong Kong
netitizens(?).
● Only 20 famous vote results are included.
● It is possible to let public to input other vote
results by hand, and submissions should be
verified by legcovotes.net authoritative.
● Including other data, eg. Minutes in plain text
or paragraphs related to a counciler.

Thank you.
● This presentation will be uploaded to
slideshare.
● Licensed in Creative Commons BY-NC-SA.
● http://slidesha.re/11ZPKvx
● http://sammy.hk

From hk0weather to Open Data, Open Government

Recommended

Recommended

More Related Content

Similar to From hk0weather to Open Data, Open Government

Similar to From hk0weather to Open Data, Open Government (20)

More from Sammy Fung

More from Sammy Fung (20)

Recently uploaded

Recently uploaded (20)

From hk0weather to Open Data, Open Government