Compare commits

...

No commits in common. "master" and "data" have entirely different histories.
master ... data

10 changed files with 144 additions and 2542 deletions

1
.gitignore vendored
View File

@ -1 +0,0 @@
baerer

View File

@ -1,72 +0,0 @@
# ccc maps
the [svg map] used on [ccc de] is at least potentially outdated
and thus in future should be updated automatically
[svg map]: https://chaos.expert/telegnom/erfakarte
[ccc de]: https://www.ccc.de/regional
## my target process
* [x] fetch addressed of a convenient source
* [x] resolve geo coordinates for addresses, e. g. [geopy]
* [x] visualize on a map, e. g. via [leafletjs] or [QGIS]
* [ ] calculate coordinates to place POI on a SVG with a known projection
* [ ] manipulate SVG so that the POI are shown accordingly even without any application
* [ ] change view port of SVG so it frames the POIs properly
* [x] create PNG from the SVG and deploy
* [ ] document for others to use
## data
### POI
the address data source is the semantic [documentation wiki], queried with [result format]
[documentation wiki]: https://doku.ccc.de/Liste_der_Erfa-Kreise_und_Chaostreffs
[result format]: https://www.semantic-mediawiki.org/wiki/Help:CSV_format
To create GeoJSON from recent address data:
1. `prepare.sh` pulls CSV from the Wiki
2. `create-geojson.sh` calls `lookup.py` for each CSV set
3. The resulting [data](https://gitea.c3d2.de/vv01f/ccc-map/src/branch/data) in [GeoJSON] format can be used with e.g. [leafletjs] or [QGIS]
### Map
The old [svg map] was nice for Germany only.
Over time more European spaces joined and the map could not display all of them.
Another point is that the projection parameters of the SVG are not known.
New material can be produced choosing the projection.
One recommended program to do this is [QGIS].
Freely useable data is available on e. g. [Natural Earth].
To create a new SVG Map:
1. Start QGIS, KBS setting recommendation: EPSG:3857 / Pseudo Mercator
2. Drop a Shape-File, e. g. sqlite format (and optionally a GeoJSON-File)
3. Adjust colors in the layers styles
4. Print as SVG (Label for POI data is lost in version 2.18)
[QGIS]: https://qgis.org/ "QGIS"
[Natural Earth]: http://naturalearthdata.com/ "Natural Earth"
[geopy]: https://geopy.readthedocs.io/
[leafletjs]: https://leafletjs.com/
[GeoJSON]: https://geojson.org/ "Website for RFC 7946"
<!--
## Thanks go to …
* [telegnom](https://chaos.expert/telegnom/erfakarte) for patiently describing the problems and needs for the map
* [ax3l](https://github.com/ax3l) for his [spontanious lightning talk](https://media.ccc.de/v/DS2016-7782-lightning_talks#t=5112) on [„Visualisierung Flüchtlingsfeindlicher Vorfälle in Deutschland“](https://github.com/ax3l/chronik-vorfaelle) at [Datenspuren](http://datenspuren.de/) in 2016 getting me started years later when I saw it again
* The [Dresden OSM Meeting](https://wiki.openstreetmap.org/wiki/DresdnerOSMStammtisch) for hinting me on QGIS and an introduction how to use it
* and quite some people helping me to finally polish some python
-->

View File

@ -1,12 +0,0 @@
#!/usr/bin/env sh
echo "this takes some time depending on the amount of addresses in your lists …"
for file in *-csv ; do
out=$(echo "${file}"|cut -d- -f1)".geojson"
if test -e "${out}" ; then
echo "file exists: ${out}, skipping."
continue
fi
echo ./lookup.py "${file}"
./lookup.py "${file}" > "${out}"
#~ ./lookup.py "${file}" 2>/dev/null > "${out}"
done

81
ct.geojson Normal file
View File

@ -0,0 +1,81 @@
{"type":"FeatureCollection","features":[
{"type":"Feature","geometry":{"type":"Point","coordinates":[10.07,48.84]},"properties":{"name":"Aalen","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[4.87,52.36]},"properties":{"name":"Amsterdam","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.14,49.99]},"properties":{"name":"Aschaffenburg","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[10.89,48.36]},"properties":{"name":"Augsburg","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.59,49.93]},"properties":{"name":"Bayreuth","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.45,46.96]},"properties":{"name":"Bern","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.53,52.04]},"properties":{"name":"Bielefeld","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[19.06,47.49]},"properties":{"name":"Budapest","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[10.95,50.27]},"properties":{"name":"Coburg","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.04,50.98]},"properties":{"name":"Erfurt","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.42,54.80]},"properties":{"name":"Flensburg","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.67,50.56]},"properties":{"name":"Fulda","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.68,50.58]},"properties":{"name":"Gießen","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[15.45,47.07]},"properties":{"name":"Graz","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.99,51.48]},"properties":{"name":"Halle a. d. Saale","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.95,52.17]},"properties":{"name":"Hildesheim","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.38,48.77]},"properties":{"name":"Ingolstadt","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.40,47.26]},"properties":{"name":"Innsbruck","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.50,53.94]},"properties":{"name":"Itzehoe","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.58,50.93]},"properties":{"name":"Jena","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[12.34,51.34]},"properties":{"name":"Leipzig","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[6.12,49.62]},"properties":{"name":"Luxemburg","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[10.67,53.87]},"properties":{"name":"Lübeck","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.40,47.71]},"properties":{"name":"Markdorf","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.64,51.94]},"properties":{"name":"Münster","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[6.69,51.19]},"properties":{"name":"Neuss","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.08,49.45]},"properties":{"name":"Nürnberg","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.94,48.47]},"properties":{"name":"Offenburg","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[13.08,52.39]},"properties":{"name":"Potsdam","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.17,51.62]},"properties":{"name":"Recklinghausen","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[12.12,49.01]},"properties":{"name":"Regensburg","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[10.18,49.38]},"properties":{"name":"Rothenburg ob der Tauber","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[4.43,51.91]},"properties":{"name":"Rotterdam","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.42,53.60]},"properties":{"name":"Schwerin","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[6.62,49.75]},"properties":{"name":"Trier","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.69,51.54]},"properties":{"name":"Unna","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.46,48.06]},"properties":{"name":"Villingen-Schwenningen","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.50,50.56]},"properties":{"name":"Wetzlar","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.73,47.50]},"properties":{"name":"Winterthur","marker":"ct"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.14,51.27]},"properties":{"name":"Wuppertal","marker":"ct"}}
]}

63
erfa.geojson Normal file
View File

@ -0,0 +1,63 @@
{"type":"FeatureCollection","features":[
{"type":"Feature","geometry":{"type":"Point","coordinates":[6.11,50.79]},"properties":{"name":"Aachen","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[10.89,49.90]},"properties":{"name":"Bamberg","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.63,47.53]},"properties":{"name":"Basel","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[13.38,52.52]},"properties":{"name":"Berlin","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.82,53.08]},"properties":{"name":"Bremen","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.65,49.87]},"properties":{"name":"Darmstadt","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.46,51.53]},"properties":{"name":"Dortmund","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[13.73,51.08]},"properties":{"name":"Dresden","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[6.78,51.22]},"properties":{"name":"Düsseldorf","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.00,49.60]},"properties":{"name":"Erlangen","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.02,51.44]},"properties":{"name":"Essen","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.64,50.12]},"properties":{"name":"Frankfurt am Main","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.84,47.99]},"properties":{"name":"Freiburg","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.95,51.55]},"properties":{"name":"Göttingen","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.94,53.56]},"properties":{"name":"Hamburg","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.72,52.39]},"properties":{"name":"Hannover","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[7.76,49.44]},"properties":{"name":"Kaiserslautern","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.41,49.01]},"properties":{"name":"Karlsruhe","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.48,51.32]},"properties":{"name":"Kassel","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[6.91,50.95]},"properties":{"name":"Köln","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.49,49.46]},"properties":{"name":"Mannheim","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[11.56,48.15]},"properties":{"name":"München","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.75,51.72]},"properties":{"name":"Paderborn","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[13.06,47.79]},"properties":{"name":"Salzburg","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.00,50.87]},"properties":{"name":"Siegen","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.18,48.78]},"properties":{"name":"Stuttgart","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.99,48.40]},"properties":{"name":"Ulm","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[16.36,48.21]},"properties":{"name":"Wien","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.23,50.08]},"properties":{"name":"Wiesbaden","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[9.92,49.80]},"properties":{"name":"Würzburg","marker":"erfa"}}
,
{"type":"Feature","geometry":{"type":"Point","coordinates":[8.53,47.39]},"properties":{"name":"Zürich","marker":"erfa"}}
]}

View File

@ -1,71 +0,0 @@
#!/usr/bin/env python3
from __future__ import print_function
from geopy import Nominatim
# ~ import json
import sys
#disable ssl verification
import ssl
import geopy.geocoders
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
geopy.geocoders.options.default_ssl_context = ctx
#/
# debug print to stderr, https://stackoverflow.com/questions/5574702/how-to-print-to-stderr-in-python
def eprint(*args, **kwargs):
print(*args, file=sys.stderr, **kwargs)
# ~ fn="ct-csv"
if len(sys.argv) == 2:
fn = sys.argv[1]
else:
eprint("in case you do not know what you are doing, better call via e.g. `create-geojson.sh`.\n");
eprint("expecting one argument: csv-file.");
eprint("received "+str(len(sys.argv))+" argument(s): "+str(sys.argv));
sys.exit(1) # exit with error
# for retreiving geo coordinates from addresses
geolocator = Nominatim(user_agent="my-mapper")
# lat: 0=111,1km,1=11km,2=1km… lng: 0=70km,1=7km,2=.7km…
precision = 2
# convert csv to geojson
firstline = True
# start json list
print ( '{"type":"FeatureCollection","features":[' )
# get data from file
with open( fn, 'r' ) as fp:
for place in fp:
#error handling: expect
arrAddress = place.split(",")
strAddress = ",".join(arrAddress[2:6])
eprint( "looking up: "+strAddress )
location = geolocator.geocode( strAddress )
if location is not None: # its a class
# todo: ceil coords to hide true location in format string
# ~ strCoordPlace = '{:s},{:.6f},{:.6f}'.format( arrAddress[3], location.latitude, location.longitude )
geojson = '{{"type":"Feature","geometry":{{"type":"Point","coordinates":[{:.'+str(precision)+'f},{:.'+str(precision)+'f}]}},"properties":{{"name":"{:s}","marker":"{:s}"}}}}';
strCoordPlace = geojson.format( location.longitude, location.latitude, arrAddress[1], fn.split("-")[0] )
if firstline == False :
print (",")
else:
firstline = False
print ( strCoordPlace )
# end json list
print ( "]}" )
# exit fine
sys.exit(0)
# ~ m = folium.Map( # etc..)
# ~ m.save("filename.png")
# ~ https://github.com/python-visualization/folium/issues/35#issuecomment-164784086
# ~ https://github.com/wbkd/leaflet-mapshot
# ~ https://stackoverflow.com/questions/44800396/python-ipyleaflet-export-map-as-png-or-jpg-or-svg
# ~ m = folium.Map(location=[51, 13], zoom_start=5, tiles='Stamen Toner') # Mapbox Bright , CartoDB Positron

View File

@ -1,34 +0,0 @@
#!/usr/bin/env python3
import math
def arc_to_deg(arc):
"""convert spherical arc length [m] to great circle distance [deg]"""
return float(arc)/6371/1000 * 180/math.pi
def deg_to_arc(deg):
"""convert great circle distance [deg] to spherical arc length [m]"""
return float(deg)*6371*1000 * math.pi/180
def latlon_to_xyz(lat,lon):
"""Convert angluar to cartesian coordiantes
latitude is the 90deg - zenith angle in range [-90;90]
lonitude is the azimuthal angle in range [-180;180]
"""
r = 6371 # https://en.wikipedia.org/wiki/Earth_radius
theta = math.pi/2 - math.radians(lat)
phi = math.radians(lon)
x = r * math.sin(theta) * math.cos(phi) # bronstein (3.381a)
y = r * math.sin(theta) * math.sin(phi)
z = r * math.cos(theta)
return [x,y,z]
def xyz_to_latlon (x,y,z):
"""Convert cartesian to angular lat/lon coordiantes"""
r = math.sqrt(x**2 + y**2 + z**2)
theta = math.asin(z/r) # https://stackoverflow.com/a/1185413/4933053
phi = math.atan2(y,x)
lat = math.degrees(theta)
lon = math.degrees(phi)
return [lat,lon]

View File

@ -1,33 +0,0 @@
#!/usr/bin/env sh
# baerer is the `authority` as in RFC 3986, here login for doku.ccc.de
fae="file already exists."
# recv data
if test -e baerer ; then
auth=$(head -1 baerer)
else
echo "error: authentication information not found."
exit
fi
#~ test -e ct-json && echo "$fae" || curl -o ct-json "https://"${auth}"doku.ccc.de/Spezial:Semantische_Suche/-5B-5BKategorie:Chaostreffs-5D-5D-20-5B-5BChaostreff-2DIs-2DErfa::falsch-5D-5D-20-5B-5BChaostreff-2DActive::wahr-5D-5D/-3FChaostreff-2DPhysical-2DAddress%3DAdresse/-3FChaostreff-2DPhysical-2DHousenumber%3DHausnummer/-3FChaostreff-2DPhysical-2DPostcode%3DPLZ/-3FChaostreff-2DPhysical-2DCity%3DStadt/-3FChaostreff-2DCountry%3DLand/mainlabel%3D/limit%3D100/order%3DASC/sort%3DErfa-2DCity/offset%3D0/format%3Djson/headers%3Dshow/searchlabel%3DJSON"
#~ test -e erfa-json && echo "$fae" || curl -o erfa-json "https://"${auth}"doku.ccc.de/Spezial:Semantische_Suche/-5B-5BKategorie:Chaostreffs-5D-5D-20-5B-5BChaostreff-2DIs-2DErfa::wahr-5D-5D-20-5B-5BChaostreff-2DActive::wahr-5D-5D/-3FChaostreff-2DPhysical-2DAddress%3DAdresse/-3FChaostreff-2DPhysical-2DHousenumber%3DHausnummer/-3FChaostreff-2DPhysical-2DPostcode%3DPLZ/-3FChaostreff-2DPhysical-2DCity%3DStadt/-3FChaostreff-2DCountry%3DLand/mainlabel%3D/limit%3D100/order%3DASC/sort%3DErfa-2DCity/offset%3D0/format%3Djson/headers%3Dshow/searchlabel%3DJSON"
if test -e ct-csv ; then
echo "$fae"
else
curl -k -o ct-csv "https://${auth}doku.ccc.de/Spezial:Semantische_Suche/-5B-5BKategorie:Chaostreffs-5D-5D-20-5B-5BChaostreff-2DIs-2DErfa::falsch-5D-5D-20-5B-5BChaostreff-2DActive::wahr-5D-5D/-3FChaostreff-2DCity%3DLabel/-3FChaostreff-2DPhysical-2DAddress%3DAdresse/-3FChaostreff-2DPhysical-2DHousenumber%3DHausnummer/-3FChaostreff-2DPhysical-2DPostcode%3DPLZ/-3FChaostreff-2DPhysical-2DCity%3DStadt/-3FChaostreff-2DCountry%3DLand/mainlabel%3D/limit%3D100/order%3DASC/sort%3DChaostreff-2DCity/offset%3D0/format%3Dcsv/headers%3Dshow/searchlabel%3DCSV/sep%3D,/filename%3Dct-2Dbesuchsadressen.csv"
fi
if test -e erfa-csv ; then
echo "$fae"
else
curl -k -o erfa-csv "https://${auth}doku.ccc.de/Spezial:Semantische_Suche/-5B-5BKategorie:Erfa-2DKreise-5D-5D-20-5B-5BChaostreff-2DActive::wahr-5D-5D/-3FChaostreff-2DCity%3DLabel/-3FChaostreff-2DPhysical-2DAddress%3DAdresse/-3FChaostreff-2DPhysical-2DHousenumber%3DHausnummer/-3FChaostreff-2DPhysical-2DPostcode%3DPLZ/-3FChaostreff-2DPhysical-2DCity%3DStadt/-3FChaostreff-2DCountry%3DLand/mainlabel%3D/limit%3D100/order%3DASC/sort%3DChaostreff-2DCity/offset%3D0/format%3Dcsv/headers%3Dshow/searchlabel%3DCSV/sep%3D,/filename%3Derfa-2Dbesuchsadressen.csv"
fi
# preprocess csv data
for file in ct-csv erfa-csv ; do
sed -e 's/"//g' ${file} > tmpfile && mv tmpfile ${file}
#~ sed -e 's/\([0-9]\{4,5\}\),/\1 /g' ${file} > tmpfile && mv tmpfile ${file}
sed '1d' ${file} > tmpfile && mv tmpfile ${file}
done
# dependencies
#~ pip3 install geopy # looking up addresses/coordinates
#~ pip3 install folium # leaflet.js with python

View File

@ -1,41 +0,0 @@
#!/usr/bin/env python3
import math
# https://tools.ietf.org/html/rfc7946#page-12
# ~ latitude = 41.145556; // (φ)
# ~ longitude = -73.995; // (λ)
# Berlin 13.38,52.52 =>
lon = 13.38
lat = 52.52
# ~ mapWidth = 200;
# ~ mapHeight = 100;
w = 1000
h = 1000
def getMecatorPicCoords( lon, lat, w=1000, h=1000 ):
"""
transform coordinates in a mercator projection
for a picture with given size
"""
# get x value
# ~ x = (longitude+180)*(mapWidth/360)
x = (lon+180)*(w/360)
# convert from degrees to radians
# ~ latRad = latitude*PI/180;
# latRad = lat*math.pi/180 # used only once, so not stored in a variable
# get y value
# def ln(x): return math.log(x) # with single argument
# mercN = math.log(tan((math.pi/4)+(latRad/2))); # latRad integrated
# mercN = math.log(math.tan((math.pi/4)+(lat*math.pi/90))) # integrated as used once only
y = (h/2)-(w*math.log(math.tan((math.pi/4)+(lat*math.pi/90)))/(2*PI)) # mercN integrated
return (x,y)
# call the function
print( getMecatorPicCoords(x,y) )

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 4.1 MiB