This looks really worthwhile and interesting, Ragnvald … worth putting this together for TZGISUG?
Was thinking of installing geonode on an amazon server. There are several projects where I would need an externally available server.
How do u unlock ur iPhone?
You will either have to buy it unlocked, or arrange with the place you bought your iPhone to unlock it for you. So-called jail breaking is also an alternative, but should be avoided.
Hi, This information is very helpful. Although, I am trying to connect to mobile internet using a usb mobile internet modem on a laptop computer. I have purchased the internet enabled sim from landmark but after 6 seconds of connecting it disconnects every time. Do you happen to know the APN, DNS and any other settings i need to connect? Or even know what my problem might be?
The disconnection issue might be related to your computer looking for a “better” network which has a higher priority. You could try to remove the other “stored” networks on your computer. As long as your modem gets you online and you can look up webpages with -non-numeric addresses you should be on a DNS. For the other settings try around. Tethering worked with “internet”, so maybe it works elsewhere as well?
Thanks for your help. Its very hard to find info for this part of the world. Do you know if buying credit is available online? The problem may be due to not having enough credit as its pre-paid, although I thought it should still stay connected. I dont have any other connections except for a wireless connection which is switched off when i try to use my mobile one. I tried the modem on another computer so i guess its a issue with the provider. I will go to land mark and see if they can help. Thankyou.
I have not heard about any options for buying credits online. You may however choose to get a monthly invoice. But not everyone can get one I assume. Your card will always start with one dollar. I have however seen that the street salesmen open the cards and empty them before selling them – as new. Nor are they any good for internet access either. Good luck Greg!
Probably not what you are looking for, but a possible workaround is to utilize Map Vendor Parameter in GeoWebCache
You will end up with many layers from GeoWebCache (one per permutation of group, species, month, etc.), but if caching is crucial, maybe it’s worth it. Maybe some species/month are more accessed/interesting than others, and worth to cache.
Is there actually two issues here?
1) Different semantic models for datasets in different organizations
2) Different WMS syntax for parameterizing queries
1 is not something that WMS can solve – it’s a business-level issue.
As for 2, a partial solution is to use a more standard way of specifying query parameters, such as encoding queries using the filter parameter and the Filter Encoding specification.
Granted, Filter Encoding is awkward and not very readable, which is why GeoServer added the ability to express filters in ECQL as well. And even then, filters are not as expressive as full parameterized SQL (for instance, you can’t express grouping as in your example). So it would be nice if the OGC was able to provide a more comprehensive standard to allow easy cascading such as your example.
1) I agree it is a business level issue, thoug it has national if not international level consequences. It might be rational for the different stakeholders to agree on it outside the WMS-standard, still as a standard. OGC?
2) My experiences from working in a shared ArcGIS and Geoserver environment is that we after starting using Geoserver are more aware of differences in implementations of WMS. However nice it is to have the vendor specific SQL parametrization from Geoserver we still would like our implementatios to be cross-platform.
Thanks for replying Martin! 🙂
Nice article and good insights. The very good thing about the open source software is its “openness”, however, we find it difficult to deal with situations in which support to the open source is no longer forthcoming and we do not have the competence or the luxury of time to do all it requires to keep the software going and our applications running. This is one of the reasons why some tend to pay the propriety owner to get it done ( for as much as it costs). Thanks anyway1 good article
For some of the more central tools/softwares like geoserver and postgis it is possible to get consultants to help out with both training and consulting.
I too am concerned about the lack of women involved in OpenStreetMap. I did a proposal to the Shuttleworth Foundation to look at the gender and geographic discrepancies within the project.
Personally I think one of the issues is though OpenStreetMap aims to be a free map of the entire world it is a highly technical community right now. That isn’t to say there aren’t technical women in the world, but the percentage is pretty low when compared to men. Not all attitudes are always friendly either. I’d love to seem more people working to change the gender discrepancy though.
I read your application – and saw the video. Both of them makes me think that your project should be funded! If not by the Shuttleworth foundation, then surely by someone.
I am thinking this because the issues you raise in full width in your application, which I just happened to stumble upon on Saturday, need further work. They need further work within OpenStreetMap, and they need further work in governmental mapping projects as well.
Good luck with your application! I will keep my fingers crossed!
Agree with Kate … this is a (admittedly poor) reflection of OSM’s origination and continued strength in open source programming communities, which tend to be male dominated. And the Foundation reflects the project as a whole … possibly even reflecting the origins even more, since it tends to be the more long standing members who get involved in the Foundation. In places where there’s been a considered effort to reach out to new places and practices … say with Kate’s work in Indonesia, or my own in Kibera, the gender balance is more equal.
The practice of OSM is distinct from and wider ranging than technical communities, and with deliberate effort, this gender imbalance can change. And the Foundation itself recognizes that it could make more effort to include new faces with its work, and it’s more challenging to volunteer for Foundation projects than it is to simply map. I’d suggest that rather than simply voting for whichever women run, we figure out ways to include more women and new people in the work of the Foundation generally.
I hate to say it, but after joining the OSM foundation on Saturday I probably tipped the balance on the gender scale even more to the male side 😉
And it is open source! Super!
At one stage I understand http://www.qdgc.org offered the possibility of downloading QGGCs for each country in Africa. This would be a very useful resource but it seems that the website has disappeared. Is this facility now available somewhere else?
I’ve started publishing the files again. Loook here: http://18.104.22.168/projects/quarter-degree-grid-cells/
Adam – I am working on it 🙂 Hopefully it will be back again by the end of this week – as a sub-project on this website.
Ragnvald – many thanks for making the QDGCs files available again so swiftly, very much appreciated.
Thanks heaps- we have been in Timor for a week and could not work out how to set up tethering. We followed yOur instructions and everything worked out fine. It took a few minutes for the personal hotspot to activate after adding the apn details but it came up OK and we can now connect to the laptop. Thanks again for your advice – it was gold!
Happy to hear it worked for you Nicki 🙂
Chris Nicholas gave some comments to this article on the SDI-Africa mailing list. This is my response to his comments:
Although Mapping Kit to the looks of it integrates a lot of the functionality it is not unique. Similar solutions exist within WordPress. The plugin does look a bit restricting, and as you can see in the data model more is needed than what I suspect Mapping Kit will be able to deliver. It is also based on MapServer.
MapServer is a very good map server platform. But to my experience it lacks a good user interface. AsplanViak in Norway has made a web portal (web user interface) called Adaptive on top of MapServer. It is unfortunately closed source software. The dutch company GeoCat is working on an ArcGIS plugin to support direct styling of MapServer (the .map-files).
GeoServer has an excellent user interface. Its integration with PostGIS through GeoNode or OpenGeoSuite shows that it is driven by a dedicated company with a sound and dedicated user base. For integration in a system like this it is of course crucial that its licens allows for redistribution and integration with other tools. I have earlier written a small text about Geoserver.
ArcGIS Server is a versatile map server with strong commercial support. The business model relies on rather expensive licenses. Licenses which comes on top of the costs of integrating it with other solutions. Redistribution of server integrating solution does not lend itself to try-before-buying. As I have also written elsewhere the money trail problem is one of the challenges in development cooperation GIS.
A national level entity, like in this case, would prefer to host a solution like this themselves if possible. The solution could be established on a physical machine, and later kept runing on in a virtual environment. Pretty much how I described it in the article Environmental Spatial Dat Infrastructure – technology.
Wordpress, indicated as the serving content management system, gives access to several plugins which would facilitate multiple languages. The data structure as indicated in the article does not. So information in maps and tables would be in one language. Adding such support is possible – but would complicate the data structure. With good context information in the wordpress pages this is bearable – albeit subject to discussion.
Handling user dialogue is not easy. Not in Norway, and not elsewhere. It requires an information strategy and also time to handle it both with regards to technical issues and content. WordPress, and other content management systems facilitate for this in many ways. Twitter, forums, mailing lists, Facebook integration and so on. National/institutional policy for the users should be determinant on this issue.
It’s a very good idea! There should probably be a mini browser for the data content suitable for use from handheld devices. WordPress has plugins which s makes it’s content readily available for handhelds, but the maps and tables would need special care. Some of it might be handled through RSS and GeoRSS. But letting the user provide positional information as a parameter for relevant geographical information is something to bring into the equation.
I have been around long enough to see my share of abandoned websites and clearinghouses. A good process will leave funding to update and refine both content and structures around it. The best way to achieve this is to have long term projects with dedicated partners. In other words this is not a technical level issue. Good projects must take into account changes in technology, methods for collecting and distributing data, politics, partner relations and a lot more. Technology is sometimes just a suitable answer at a fixed point in time. Good solutions are dynamic. And sometimes the best solution can even be to pull the plug of a system.
Can anyone tell me how to setup the APN? I just bought a tablet pc which said can connect to 3G network (either use simcard directly or through modem) however it does not work. Any advise is higlhy appreciated.
Worked for UNMIT and IOM in Timor-Leste in 2008/9. Found reading this very interesting, and also very familiar. Actually, recognise the UNMIT GIS Chief Timur in one of your photos. Anyway, with the UN Mission mandate coming to an end at the end of this year?, and with the UN Mission having such a wealth of information, will they then contribute/ organise/establish a body/ organisation that can take the building of geospatial information/ development of standards/ skills further?
I need some information about climate change then about relation with pollution
Dear Pedro, I know there are metereological data available. With long enough timeseries I assume those would be relevant for climate change analysis. For pollution data I am afraid I do not know about any such data. In a many countries one would find agencies monitoring the environment using chemical indicators for pollution levels. I am not sure if this is the case for Timor Leste. I would suggest you contact DNMA in the Fomento building for further information.
I attempted to download the Quarter-degree-grid-cells (QDGS) for Africa from your website; unfortunately the link appears to be broken as I get the following message: “page not found”.
Any possibility of getting this file? I am mapping some plant distribution and the data available are on a QDGS basis.
Thank you very kindly,
Hello Roland – please note that there has been updates to the QDGC files on my website. QDGC for Africa now covers levels 1 to 4. They are available for download here: http://22.214.171.124/download-qdgc-continents/
Thankyou for pointing this out Roland. The link above did not work, although the link in the menu did. Fixed it and now you should have access to it from this page as well.
I am interested in knowing more about the use of the QDGC, so if you find the time I would appreciate knowing more about your project.
Did an update on the article. Please consider if your comment is still valid.
Nice one Ragnvald! Dissolving 7283 is a big job and it good to hear your work-around solution. But why does dissolving polygons in groups of 10 take longer than dissolving them in groups of 110? I would have guessed small groups take very little time and large groups take longest. Weird??
Only ESRI would consider a dissolve of 7000 buffer polygons “large”!
You might like to have a look at JEQL, which is built on the JTS Topology Suite library for geometry processing. It provides a flexible and performant way of running spatial operations on datasets.
I ran the agder_buffer.shp file through a JEQL script to union all the polygons, and it completed correctly in 1.5 sec. The JEQL script to do this is:
ShapefileReader t file: “agder/agder_buffer.shp”;
t = select geomUnionMem(GEOMETRY) g from t;
That’s fast Martin! Is the JEQL script a library available under Python?
JEQL is really its own standalone language and engine at the moment. It runs on the JVM, but really that’s invisible to the user, since they never interact with Java directly.
For integration with Python I’d recommend checking out Sean Gillies’ work on Shapely. It uses the GEOS C library as it’s geometry engine. GEOS is a C port of JTS, so it provides similar functionality (and close, but not quite as good, performance).
Wow … 1.5 sec is incredibly fast!
I also just tried this in QGIS (dissolving all the buffers at once) and it did it in ~ 41 seconds (OSX 10.7, QGIS 1.8, 8GB RAM and SSD). It’d be interesting to adapt the script to run in QGIS and compare your results!
YES! The dissolve process is sub-optimal. I’ve encountered issues with the dissolve operation since ArcGIS v 9.0. I haven’t had the opportunity to do any testing since 10.1 – which given some extra time might be worth doing as it’s been a thorn in my side for sometime. The issues I’ve encountered haven’t so much been on the process hanging but the introduction of splits in the dissolved geometry. Dissolving geometry over a large extent with a large number of features actually introduces split features. For example, a polygon that didn’t have another feature to dissolve with might end up becoming 2 features. When these split features are looked at, it is obvious there is some tiling schema that introduces these splits resulting in geometries not being dissolved but actually split. Manually splitting up large data sets is one way around it, but is this really acceptable for some basic functionality? Even a feature class with 1000s of records and many overlapping polygons should be processed as expected, but as with ANY ArcGIS tools, careful examination of results is always warranted. Curious about the alternatives to ArcGIS suggested, maybe someday I can devote the time to explore. Thanks for the tips!
I’m not sure how you would go about “adapting the script”, but I believe QGIS uses GEOS for geometry processing, and GEOS has the same cascaded union functionality as JTS (which is what JEQL uses). So you could probably get this to work in the same way. Perhaps GEOS is exposed to QGIS Python?
Your unary union is speedy. On my old laptop:
>>> from fiona import collection
>>> from shapely.geometry import shape
>>> from shapely.ops import cascaded_union
>>> def dissolve(c):
... ta = time.time()
... u = cascaded_union([shape(f['geometry']) for f in c])
... print time.time() - ta
... return u
>>> c = collection("/Users/seang/Downloads/agder/agder/agder_buffer.shp", "r")
>>> u = dissolve(c)
Well, it is a pretty fast machine – 3.4 GHz 8-core (although the process is single-thread only).
On my creaky old 2 GHZ machine I get about 4.5 sec.
The difference may be the performance penalty for C memory allocation in GEOS, as opposed to Java’s highly optimized memory managment.
I meant that I think 8 seconds is fast considering that my laptop is 4 years old. And on top of the slow memory allocation Fiona reads and formats all attributes to JSON by default (I didn’t read just the geoms).
Right, the main thing is that they both blow the doors off Arc!
Still memory-bound, though – but I have some tricks up my sleeve for dealing with that problem…
I encountered problems with Dissolve_management recently while trying to convert a python script that worked in 9.3 into one that would work in 10.1. The dissolve worked fine in 9.3 but crashed python with the 10.1 version. I then tried the dissolve using ArcMap and the Arc Toolbox and it locked up giving me a dialog to report the bug to ESRI.
The thing is that I was working on a Virtual XP VM environment with limited RAM (4GB). The dissolve worked for others testing for me on better machines with the same layer (only 812 feature in my case) so I’m still looking into it. Thanks for the insight. This may be useful for me.
I have uploaded an example using multiple (4) processes at once using FME Desktop 2013. For the record this takes 4.1 seconds on my computer on the Nordland dataset and took 4 minutes to create….
Files and workspace can be downloaded here. http://depositfiles.com/files/tcxuedugu
Looks like there are more people having this issue.
Så fint at dere deler med oss 🙂 … og bloggen blir jo en fin minnebok for dere selv også. Gleder meg med dere!!!!
Takk for flotte reisebrev. Vi gleder oss med dere. GUNHILD
Takk for fine reisebrev! Gratulerer med dagen til Synne!!!
Thanks for your time and effort looking at the Geoprocessing Dissolve tool in 10.1 SP1. We’ve been working hard on this and other tools to improve their performance.
I’m happy to inform you that for 10.2 we continued to work on the performance of the Dissolve tool to maintain the quality of our output while at the same time dramatically improving the performance of cases such as the ones you used in this post. When running a Dissolve all operation in ArcGIS 10.2 against the data you provided we are seeing the following performance of the Dissolve tool:
(Specs – Win7 64; 2xDual Core 3.14Mhz CPU; 8GB RAM)
Dissolve ALL Time
agder_buffer.shp Under 7 seconds
nordland_buffer.shp Under 6 seconds
If anyone finds a Geoprocessing tool that they feel is not performing as expected against their data, please contact us directly and if possible share it with us so we can do all in our power to make things better.
Thank you very much for your feedback! As always I will be looking forward to the next version of ArgGIS. ESRI products are central to the work I am doing and much of it would not be possible without 🙂
I would of course also urge you to use and include relevant open source libraries in your software – both for the benefit of its users, ESRI and the other communities supporting open source software!
Hi there Ragnvald,
This is neat stuff. 🙂
You might be interested in Geoff Dutton’s work on a quad-tree geoaddressing system called the octahedral quaternary triangular mesh (O-QTM). Here’s a link to a relevant paper: http://www.spatial-effects.com/papers/conf/GDutton_SDH96.pdf
Thanks Martin 🙂 I had a look at Geoff Duttons paper upon your request. It looks nice – BUT what I am not seeing (correct me if I am wrong) is a naming convention that could work with rangers and other contributors to field data. I am sending you our paper on Quarter Degree Grid Systems in a separate mail for more information.
Hey, I don’t see any maps, or links to maps on this page…
Just checking out your blog.
Got a point there Hubbard. Will look into it. Guess the planning of the site got the best of me and I just forgot about it. But there are several maps which could be posted.
Note that the QDGC range of files published by me now covers all states in the world. They are rather useless for countries above 60 degrees north though – as the squares become quite elongated. I am thinking of solving that by publishing a version which splits them in half at a certain latitude providing for a square system (sort of) further north.
I am so delighted to have found the resources you’re offering! I run an Africa wide mammal conservation project and am doing a lot of analyses of the spatial data. All the data I have are at GPS level and (for multiple reasons) I would like to scale these up to QDGC level. I see from comments on your website that you may have an Africa wide QDGC GIS layer, but I’m struggling to find this. Can you please point me in the right direction?
Also, please let me know how best to credit you in the analyses I do.
Hello Tali – good to be of help 🙂 I will try to prepare a Pan-African QDGC-file by tomorrow. It will consist of QDGC levels 1-4. Level 2 only is available for download here:
For credits just use the article we wrote some years back as a reference in papers or similar:
Thank you so, so much! Level 2 is actually the one I need for now, but I’d be happy to see what the other 3 levels look like when you get a chance to prepare them. Please dont rush on my account!!
Thanks so much,
Hello again Tali – QDGC 1-4 available for download. An update on the per country files are on their way 🙂
Continents and per country files of QDGC levels 1-4 are now available for download 🙂
Wow!! Thank you so, so much! I can’t believe how much work you’re willing to put in for the good of the rest of society! Thanks again 🙂
Thank you so much!!!! I had no problems tethering when I was here last year, but this time it wouldn’t work – so glad I found your solution! 🙂
Nice interesting article I have just read from this website. Well, I was working with UNMIT as Meteorological Officer 2011-12 and had a first hands experience towards these drastic problems of data management and other sensitive sectors. Such sectors involved climate change, pollutions, ocean coastline environment management, plus many more. There is data available available but only for only UNMIT duration only. In between these data sets, there are some relapses in between but better than nothing. This article hope it touches to pretty serious actions of implementation soon as UNDP’s climate change unit take the cue to develop and research this sector to exhaustion. Thanks
Hi, I seem to be struggling to download the QDGC for Africa. I follow the the tab “Projects” then “QDGC” and “download QDGC-continent”. I then click on the “Africa QDGC 01-04” however once the page loads there are no further links only a tiny smilie face. Please help! And thank you for such an incredible data base!
Agree that there is a problem. Will look into it. In the meantime, please use this link: http://126.96.36.199/download/QDGC/continents/africa_qdgc_01-04.7z
Fixed it. It was an issue with access rights in the file management plugin.
This article may be of interest – highlighted in the July 2014 GSDI Regional Newsletter, http://portal.gsdi.org/files/?artifact_id=1534
Applying the theory of planned behavior to explain geospatial data sharing for urban planning and management: cases from urban centers in Tanzania
Author(s): Alex Lubida, Petter Pilesjö, Margareta Espling & Micael Runnström
African Geographical Review, Published online: 17 March 2014 [not an open access journal]
Abstract: This paper illustrates the potential use of the theory of planned behavior as a guiding framework for understanding intentions and behavior in geospatial data sharing in Tanzania. A structured questionnaire survey, was constructed and sent to local governments as well as academic and private organizations that are major
producers and/or users of geodata. The questionnaire covered issues of how collection of geodata is financed, management of geospatial data, and compatibility of spatial data-sets. The theory was found to be generally effective in accounting for intentions to share geospatial data in Tanzania (p < .01). Results show potential for data sharing between local governments and other organizations in Tanzania, and it is suggested that creating awareness among spatial data stakeholders and the establishment of a spatial data infrastructure policy framework will speed up geospatial data
Thanks for the article, it summarizes what Thomas and Shane covered in the workshop.
The workshop was truly beneficial to us who attended it.
During WW2 or shortly after, my name was sent with a Jr. Red Cross box of gifts to a family on Mindland Island, Norway. If I remember correctly, their last name was Norberg. There were three children, Turid, Gunnar, and Anny Margaret. We corresponded for a while, then lost touch with them. I am wondering if any of them might be still living there. Is it possible to find out? Thank you
Hi, Beth Dhuse.
I am Viktor Gjersvold, son of Anny Margrethe. We are still living on Mindland.
Gunnar and Turid, my uncle and aunt lives not very far from Mindland.
I would like to send you the “Norway letters” and pictures that we saved from your family in the 40’s. My sister has them now, but I’ll send them as soon as I get them back. I’ll need an address. Thank you – Beth
Please could you conduct a training on analysis of wilderness. I hope that will help us wildife division of Tanzania
As mentioned in the email I am thinking that the TAWIRI-conference could be a suitable place to make a presentation of such work. Depends a bit on this years theme though. TAWIRI tells me they are initiating the program work these days.
Thanks so very much for this!
Good to hear it is of value and interest 🙂
Hai Ragnvald Larsen,
thank you for this wonderful information.
how to install shape_to_poly package.
after downloading there is no .exe file ?
The code runs under python, so you will have to install python on your computer. But I think the software missionplanner now supports this so the code is thus redundant 🙂
Are you saying that mission planner 1.X import shape files?
After downloading the file should I change the ending to .py?
Open Mission Planner an press Ctrl-F. This will open a page with buttons. One of them is Shp to Poly. So it works out of the box 🙂
For the country Tunisia, level 3 doesn’t work.
Thank you for your feedback. I will look into it, but can not promise any updates shortly. Try out the regional data sets (africa) in between, you might be able to find working files there.
Thanks. But there is no level 3 for Africa.
Hei Rangvald, takk for link til interessant artikkel. Jeg har distribuert denne videre til kolleger som jobber I Tanzania. Dere har nok like erfaringer. Lykke til videre, du er med på noe spennende!
Very useful, thanks for making this available
Thanks for the good work but am failing to access the map of the African continent. Keep getting the message “page not found”. Any advice on a way forward?
The following link should work: http://188.8.131.52/wpfb-file/qdgc_africa-7z/
ESRI’s implementation is definitely very slow. I’m not sure why. I was trying to do the buffer and dissolve all using ESRI and had to implement Shapely’s libraries. I was taking hundreds of thousands of points and buffering them and then dissolving them (so 7000 is nothing). I did a quick test recently and about 11k points took about 3 minutes, 59 seconds using ESRI’s buffer function and through my shapely implementation, it was 13 seconds.
So, I tried using the arcpy Geometry class to see if I could speed it up and using the Geometries individually and using the union function actually took way longer than the built-in buffer. The buffer seemed as quick but the union was a lot slower. I don’t think there’s any way to speed up ESRI’s implementation.