En god del av de tingene jeg skrev var, om jeg skal si det selv, ganske bra. Men det ble aldri helt ferdig – og jeg fikk andre ting som interesserte meg mer. Så jeg lot det bare ligge, helt til i dag. I dag har Fornyingsminister Rigmor Aasrud (Ap) og miljøvernminister Bård Vegar Solhjell (SV) satt i gang et arbeid med en plan for frigivelse av kartdata fra det offentlige i årene som kommer. Planen skal presenteres om kort tid, rundt sommeren, sa Solhjell i en uttalelse.

Det må bemerkes at innlegget ble skrevet i god tid før OpenStreetMap så dagens lys.

Kartverket og markedet

I 1773 startet Norges Geografiske Oppmåling den første offisielle kartleggingen av Norge. I 1785 ble en kyst- og sjømåling iverksatt av samme organisasjon. I dag, snart 230 år senere, jobbes det fremdeles med kartlegging av Norge. Motivasjonen for kartlegging er den samme i dag som den var det på slutten av det syttende århundre; menneskers sikkerhet og sikring av økonomisk verdier.

Det er dette grundige kartleggingsarbeidet som gjør at du kan legge ut på fjelltur en sen høstkveld i trygg forvissning om at du vil komme frem til bestemmelsesstedet. Seilturen langs norskekysten med gode sjøkart gjør at vi kan fokusere på turopplevelsen. Kart er en forutsetning for sikker transport både til lands, til vanns og i lufta.

Nabokrangler om eiendoms- og bruksrett kan ved hjelp av tinglyste skjøter med kart avklares uten rettslige forhandlinger. Uten det økonomiske kartverket som dekker byer og tettsteder ville en oversiktelig arealplanleggingen på lokalt nivå også vært en umulighet. Kart er en viktig del av den økonomiske hverdagen vår.

- – - – -

Ansvarsområdene for kartverket omfatter Norges land-, kyst- og havområder og kysten rundt Svalbard. De viktigste oppgavene for kartverket er å etablere og drive en nasjonal infrastruktur av geografisk informasjon. Denne kalles i dag “Norge Digitalt”. “Norge Digitalt” er ifølge kartverkets egne websider et felles fundament for verdiskaping som skal bidra til sikker ferdsel. Et annet mål er at offentlig forvaltning og næringsliv skal ha et verktøy for å løse sine oppgaver mer effektivt. Kartverket er i dag en forvaltningsbedrift med Miljøverndepartementet som fagdeparte­ment.

Statens Kartverk har skilt ut en egen enhet som driver med salg av trykte og digitale produkter, oppdrag og konsulenttjenester basert på datagrunnlaget i “Norge Digitalt”. Denne virksomheten finansieres i sin helhet av salgsinntektene og er skilt ut i en egen divisjon.

- – - – -

For å lage kart i Norge må du ha tilgang til geografiske data. Det fins to ytterpunkter for prising av geografiske data. I USA har man ført en aktiv politikk for å gjøre føderal (statlig) informasjon lett tilgjengelig i elektronisk form. Konsekvensen er at slike data er tilnærmet gratis i USA. Tilgangen er svært enkel via internett. Det betyr at hvem som helst kan laste ned detaljerte geografiske data og bearbeide dem for videre salg. I England har staten mer eller mindre monopol på geografiske data, og dette benyttes for å sikre dekning av utgiftene til å hente slik informasjon.

I Norge har man valgt en mellomting mellom disse ytterpunktene. I praksis betyr det at staten går inn med en midler slik at virksomheten til en viss grad subsidieres, men at brukere av digitale geografiske data likevel må betale for å få tilgang til slik data. Rettighetene til kartgrunnlaget eies av staten.

- – - – -

Geografiske data er vår tids originalkart. I det private kartmarkedet mener man at dagens prispolitikk for geografiske data er diskriminerende. Det blir, sannsynligvis med rette, hevdet at avstanden mellom kartverkets salgsdivisjon og “Norge Digitalt” er for liten. Kartverkets salgsdivisjon kan som en konsekvens av dette jobbe med forretningsideer uten å måtte stille kapital til rådighet. De kan også nyte godt av god tilgang til informasjon om utvikling av “Norge Digitalt”. En annen interessant faktor er det at uttak av kartdata formidles gjennom kartverkets salgsdivisjon. Dette er en organisasjon som i praksis er en konkurrent til de private aktørene. Samlet er skaper dette en urimelig forskjell mellom kartverkets salgsdivisjon og de private aktørene.

På den ene siden fører dette til at kartverkets salgsdivisjon kan få fordeler som er konkurranse­vridende. På den annen side kan dette føre til at “Norge Digitalt” lekker ressurser til sin egen salgsdivisjon, og dermed tapper ressurser fra kartverkets primæraktivitet som er å lage et kartgrunnlag for norske områder.

For å kunne ha en sunn konkurransesituasjon bør tilgangen til de kartdata som er en del av “Norge Digitalt” være tilgjengelig på lik linje for alle som opererer i kartmarkedet. En forskjellsbehandling kan være alvorlig i forhold til nyskapning i en voksende, men sårbar, digital kartnæring.

- – - – -

Til syvende og sist dreier kartverkets eksistens seg også om hvorvidt kartleggingen av Norge skal være en offentlig oppgave. Kartverkets budsjetter indikerer at kartleggingen av Norge er mindre prioritert nå enn tidligere. Regjeringens budsjett for inneværende år innebar en krymping av kartverkets budsjetter på 38 millioner – altså nærmere ti prosent. Betyr dette at politikerne ser et mindre behov for kartlegging nå enn tidligere? Tallenes tale skulle være helt klar.

Politikernes prioriteringer kan imidlertid omgås. Et økonomisk presset kartverk kan kanskje tvinges til å benytte seg av den nære koblingen mellom “Norge Digitalt” og salgsdivisjonen. Det hele selvsag i beste hensikt. Ressursstrømmen fra en salgsdivisjon med store fordeler i markedet kan dermed brukes for å hente penger inn i driften av kartverket. Det kan gi oss et kartverk som i mye større grad er styrt av markedet. Hva vil det bety for de økonomiske verdiene og den sikkerhetsmessige siden av arbeidet med kartleggingen av Norge?

Den nære koblingen mellom “Norge Digitalt” og kartverkets salgsdivisjon kan til syvende og sist gjøre at politikernes prioriteringer ikke når fram. Det kan forskyve ansvaret for den utviklingen vi måtte få fra politisk til markedet. Det er kanskje greit når vi snakker om kart benyttet i den økonomiske hverdagen vår. Men hva da med sikkerheten?

Vi har i de senere år sett viktigheten av å ha gode kart. En samordning av planlegginsarbeid ved hjelp av oppdaterte digitale økonomiske kart kan forenkle planprosessene både for det offentlige og de private aktørene. Man regner også med at forbedrede sjøkart kan redusere antall ulykker langs kystem ned inntil 40%. Det kan med andre ord bli dyrt og ansvarsløst å ikke prioritere arbeidet med kartlegging av Norge.

- – - – -

I dette vanskelige landskapet skulle man gjerne hatt et kart. Vi får håpe at politikerne har et slikt kart, og at kjentfolk har fått gi innspill i kartarbeidet.

Jeg ville nok ha skrevet det litt anderledes om jeg skulle ha publisert det i går. I dag ville jeg ha skrevet det på en helt annen måte – for i dag ser det ut til at landskapet er i ferd med å endre seg

This post is password protected. To view it please enter your password below:

Password:

This post is password protected. To view it please enter your password below:

Password:

This post is password protected. To view it please enter your password below:

Password:

This post is password protected. To view it please enter your password below:

Password:

The Sentinel 2 satellite modelled by RAMA, published via Wikipedia

The ESA Sentinel 2 satellites will provide the global community of environmental scientists and managers with fantastic terrestrial multi-spectral high-resolution optical data. ESA will give the general public and partners with access to these data sets. The respective users/countries will then have to do some processing of the data sets to render them useful.

In this posting I will try to present some of the work I did as part of  a national level working group last year. I will also indicate some of the challenges ahead of institutions working with environmental data management in view of the Sentinel 2.

Challenges include establishing relevant operational products, coordinating such processes and making sure that time series of the same data are available. The posting is mostly based on our report to the Norwegian Space Center.

Highlights from ESA tells us that Sentinel 2 will offer a rather nice cocktail of data based on the following capabilities:

• Multi-spectral information with 13 bands in the visible, near infra-red and short wave infra-red part of the spectrum. In part overlapping with Landsat 8.
• Systematic global coverage of land surfaces : from 56°South to 84°North, coastal waters and all of the Mediterranean sea
• High revisit time: every 5 days at equator under the same viewing conditions. Not too bad in Europe either
• High spatial resolution at 10m, 20m and 60m depending on bands
• Wide field of view: 290 km

In addition to delivering data according to the specifications the ground segment will be providing the end users with a steady flow of data. When the regularity and quality is predictable we have the basis of w what can be called operative products.

From the perspective of a geographer working with biologists and environmental managers this poses both opportunities as well as challenges. On a professional level I would like to select relevant data and make products which provide the users with both current views of the environment and long time series. All of it after good processes involving multiple stakeholders. After this I would rather someone in a central agency magically deliver the data sets to me according to our specifications and upon request as well as store them for me for later or long term use.

Understanding satellite related products

Last year I was part of a work group doing an analysis for the Norwegian Space Center on how to prepare the reception and application of optical sensor data for Norway Digital – the national geographical infrastructure. Norway Digital is the Norwegian government’s initiative to build the national geographical infrastructure. Norway Digital has since 2005 been a working co-operation and infrastructure with reference to available data and thematic data covering more than 100 operational web map services, geoportal and other services. (source: Norge Digitalt).

In any case, the report we did (in Norwegian) is available for download here:

One of the outcomes of the report was to specify national needs, particularly in the Norway Digital co-operation and assess current monitoring and surveillance programs. Part of this work was to go through older product surveys and also to arrange a workshop. In short we wanted to understand how Sentinel 2 imagery could be the basis of operative for Norway Digital partners. To be operational services should be continuously available with proven quality and regularity. The Sentinel 2 satellite is a good start. But it will require us to establish a chain of products relying of many operations – even after having the data delivered to our national doorsteps by ESA.

The below figure (1) shows how the Sentinel 2 data through a chain of processes and product definitions finally ends up as end user products.

Figure 1: Information flow

One findings from the report was that institutions, offices or persons representing the user level does not necessarily know how to describe their own product. Furthermore – when they did try to describe a product we some times found that they were describing a product used by a neighboring institution – where the only difference is that they are using a different terminology.

We set out to understand what products where out there and came up with a lot of products we in turn tried to relate to each other. What we got back was initially a rather messy relation diagram. Still – we saw that there was a pattern in there. The below figure is an illustration of product independence based on our questionnaires and  former product surveys.

Figure 2: Product interdependence

We used yEd graph tool to help us see how the products related. Where we found that there were minor differences between the products, or we assumed they were actually the same products referred to with different terminology, we merged them into one. It was a coarse process, but all we wanted was an overview, so we thought it would be ok. We could always pick the brains of the experts at a later point in time.

More structure, some merging, and this is what we got (figure 2). Satellite products are yellow:

Figure 3: Products and relations

Using our trusty yEd we took a quick look at which products were more important than others. yEd allows for a weighted presentation and using that functionality we got an interesting visual presentation:

Figure 4: Weighted presentation of the different products importance (source/derivative analysis)

The referred report contains a table of products sorted according to how many derivative products are under a particular product. The big orange square in the figure above is a vegetation index product. Derivative products of the vegetation index are for example forestry related products, vegetation change index and more. Our conclusion was that a more structured approach is necessary to be able to understand how a national community of spatial data users (government and others) can find common ground for common products. We will get there, after a few workshops.

What lies ahead?

Where we used to have project oriented initiatives we will in the future hopefully have operative products. No longer products based on one-time funding initiatives, products based on long term funding allowing us to establish and maintain recurring versions of good basis products.

At a point involved institutions will have coordinated their own product needs enough and started to prepare products based on joint efforts. Some products will remain on a national level, while other will have to be coordinated on a lower level. As indicated in the start of this posting it will be a huge undertaking. Not only will it require us to develop and fund capacity to prepare our products, it will also require us to prepare and fund storage strategy for the same products.

Where we nowadays have systems for establishing, storing and disseminating vector based data we will in the future be expected to expand that capacity for vector data – and establish systems to handle pixel and scene imagery data.

Even if we are using vector data derived from satellite imagery both research and management institutions will still have to support availability of data for verification of scientific or management conclusions. Building a national spatial data infrastructure for satellite imagery will require great efforts. Interacting with such databases from a stakeholder level will require us to keep up to speed or even develop standards for satellite storage and data exchange.

If we add Landsat 8 and other relevant imagery products to equation we have a good starting point for many relevant products for understanding and managing our environment.

It is my guess that OGC will play a role in this work, and perhaps are they already involved in related work. Bringing satellite data to users will require good specifications from the ESA delivery mechanism to user level products. This will require excellent systems for communicating the specifications and it will above all require a great ability to compromise when making them.

In other words – if you are working with GIS and environmental data change is coming – in a couple of years

Until then we will have to find answers to these questions and more:

• How should national level pixel level data be stored?
• What meta data should accompany this data?
• What protocols are relevant for transfer of data?
• Are there any standards we can draw upon?
• Who else are working on these issues in relation to Sentinel 2 imagery?
• Best practices?
• Which stakeholders should be involved in this work?
• References to advice on how open source software could play a role in this work are most welcome.

If you know the answers to some of these questions I would like to hear about it. I am curious. Use the comments fields on this blog.

If you find errors or things which are incorrect you are also welcome to use the commenting field below.

As part of the analysis I ended up doing buffers around islands in the Norwegian archipelago. After which it would be necessary to merge and dissolve the objects. This turned out to be problematic. But for some of the shapefiles I was working with ArcGIS (arcpy and python) simply failed to complete the dissolve operation.

After contacting our local ESRI representative, Geodata AS in Norway, they concluded that this was related to the following error in ArcGIS 10.1: NIM079373: Running a large number of features through the Dissolve or Buffer with dissolve option, hangs during process. I have not found any publicly information with this reference.

One could say that 7283 polygons is a tall order. One could perhaps also say that working with polygons in a task like this rather than with raster is asking for problems. Given enough time I will look into it – later – in that quiet week when nothing else is going on at work, sometime.

This blog post is about but  how I came to understand more about the limitations and possibilities with the ESRI arcpy Dissolve_management tool. It is also explains how I found a rather surprising way to make it faster.

Buffering around n thousand islands in different regions and then dissolving them to one object works fine most of the time. But for two of the regions the Dissolve_management just stopped processing. The same thing happened if I tried doing the same thing from ArcGIS desktop.

Since I really had to make this work, I tried different ways to fix it. Googling gave me some answers, but my dissolve still hung. So I divided the input file objects file in two halves after which I again tried doing the dissolve operation. Both dissolve operations succeeded. I then continued to merge the resulting files before doing a dissolve operation again. It worked. So somewhere in that fuzzy code within ArcGIS (arcpy and desktop) there is a tripwire stopping the operation.

To handle this in general I wrote a function which splits the input file into smaller files grouped by given number. the objects in these files are dissolved. The resulting files where then merged to one file and dissolved.  And just to remind the reader – I was still using the  arcpy.Dissolve_management function. The figure below explains the procedure:

Divide and conquer dissolve procedure

Running the “new” dissolve function I noted that the time the whole process took varied. I initially expected the whole process to take longer than it would using the ordinary functionality.

It turned out varying the group size had a rather big impact on the time the process took – in a positive way.  I got curious and added timers around all dissolve operations in the script. My expectations for a lower performance were not met. The new procedure was faster. I also prepared at batch script doing the operation on the same input file with a group size varying from 10 to 600.

The result based on nordland_buffer.shp was interesting and to make sure this was not only about that one file I did an additional analysis on a similar file (agder_buffer.shp). The below figure gives you the general idea of what happened:

So what’s in the bottom of the curve? It looked like there is a minimum time for the job when the objects are grouped at a certain size. I was right…

That’s it… the minimum time for these job is when the objects are in groups of around 80. The dissolve operations only takes from 25 to 28 seconds for the tested files. The required padding (merge and delete operations) around the essential dissolve processes adds marginally to the time used.

How does this compare to using the dissolve function straightforward without using above mentioned function? The Nordland dataset hangs, but fortunately the Agder dataset runs through. The total time used for dissolving the dataset is 149,5 seconds!

Could I be missing something here? Or is it the dissolve function from ESRI a rather sub-optimal one?  And that it can be made more efficient and stable by simply grouping the input file in sizes of around 80 objects?

There are a host of reasons which could confuse the above picture. Here are some of them:

• The number of overlapping polygons in the input files can make a big impact. So the issue with a sub-optimal dissolve function might not be relevant for an input file with fewer overlapping polygons. The example is extreme with +50 overlapping polygons.
• The remainder of the total polygons divided by group size will vary. This might have implications on the total time to perform the dissolve operation.
• The files I have used might be very unorthodox.
• The computer was in use while the calculations were made. Other activity might have influenced on the time used.

Since working through this problem in December last year I have happily concluded my alien species project, and as such this issue is not of my concern anymore. If someone finds the above of interest I would be curious to have some feedback. I am always eager to have arcpy script go faster. I am of course also interested in other approaches using for example open source libraries.

If my assumptions hold water I suggest that the ESRI guys and girls sit down and remake their dissolve function. It is basically sound, but something is amiss. And when the original function hangs they should give the user some feedback about this. How difficult could it be to implement a failsafe?

To allow for further testing and experimenting I have included the files used in this little experiment:

I would also like to point you to the following discussion on http://gis.stackexchange.com/:

• How to divide a feature file into ranges

At last I would like to thank my colleagues Johan Danielsen and Martin Bartnes for  their contributions and help in the process of understanding the shortcomings of the dissolve functionality in ArcGIS.

Military drones 2005

The term drone is usually associated with unmanned planes used for military surveillance and aggressive activities. The latter is something we as conservationists, scientists, environmental managers and others would rather not be associated with. Surveillance is a term slightly associated with our own trade. We call it monitoring – but much of it is about the same thing. We want to know who is where, how many they are and what they are doing. Our purpose for monitoring the environment is to be able to understand a certain species, their interactions with other species or its immediate environment.

Quality information is necessary to make decisions to protect the environment, or to provide politicians with a basis for policy decisions.  A question which has surfaced is: Will drones make conservation and management of the environment more efficient and accurate?

In this posting I will take a brief look at some of the challenges and opportunities pertaining to the use of drones in conservation.

Reading about drones and considering it for conservation use I have become somewhat of an addict. I find myself looking at prospective planes very much the same way as I looked through catalogs with RC planes some 30 years ago. Back then the planes did not serve any purpose. And I never had the money to buy one either. And what is the point in driving a plane around just for fun?

Today my ambition is to press but one button and the plane does the rest of the job on its own. I would rather blame a crash on bad programming rather than explaining how I trimmed the rudder and what not – leading to an very likely crash.

One of the planes which look like they could be a good starting point is a plane fro DIYDRONES. Quadcopters is the newcomer which is basically a helicopter with four rotors. This makes the vehicle flexible with regards to maneuvering. The software running these small birds is becoming more flexible as the days and months go by. An open source project is the Arducopter software.

To get up to speed on the options out there I would suggest the following sites for further reading:

• conservationdrones.org

• Orangutan conservancy
• Conservation magazine

So in an initial article I would like to list up some issues which should be addressed with regards to drones in conservation.

National regulations will both guide and limit your use of drones. Particular care should be made when you operate in a country where you are a foreigner.

• National remote control radio frequency regulations
• Aerospace regulations (and coordination with tourism related manned flights)
• Security regulations – governments tend to be sensitive towards equipment which might compromise military operations or areas

Pollution is an issue which concerns every person initiating activities in nature. Protected areas subject to the use of drones is no exception.

• Drones will be missing in action. The ideal is to track them down. In case they are lost it is of importance what materials used. Wood is I guess better than hardened foam.
• Some of the electrical components represent
• Wildlife interference (birdstrikes, disturbance, pollution)

Methods development should be started outright. It is of key importance to be able to use current methods and also use this opportunity to agree on standards for storage and dissemination of data.

• Projects should be established to test out replacement for aerial surveys (counting animals: http://www.african-elephant.org/tools/ctganimen.html) - scientifically sound methods and standards should be considered.

Data licensing is an issue when you collect data. Biologists are in particular good at keeping valuable observation data in the bottom of their drawer, just in case they get their act together and finally write that paper. Don’t go there. Put a license on your data and make it publicly available when relevant and possible. Remote sensor based data is no exception. The Landsat program has provided important data for many, many years. It is  my humble opinion that clear open licensing will be to the good of the environment.

• Consider a licensing system for imagery so that images can be made available in a central repository for OpenStreetMap or other use (misplaced imagery is a problem: http://www.tzgisug.org/wp/historical-aerial-imagery-in-tanzania/ )

Data availability with proper licensing is the best starting point for conservation and research.  At this point there are no central repositories where licensed aerial imagery can be stored with proper metadata. Too many times have I come over old imagery with no licensing information. Using such data could get you into trouble.

• I am suggesting that aerial imagery is established using a standard based on agreements in the conservation community.

I would be more than happy to get comments on this article. Good ones will be credited and incorporated in the article.

Last time, in March, I had decided enough was enough. I wanted to get on line in Timor Leste, but it took some efforts to get through with what was the plan. I am now here for a third time. With past experiences it took me less than an hour from landing at the airport until I had a new micro-sim-card in my iPhone.

This is an update of an article first posted feb 29th, 2012 @11:26

First I bought two sim-cards from one of the “pulsa” (credits) sellers. The SIM-cards were normal sized sim cards, and I managed to cut the first one in pieces rendering it useless. The second one fit in my micro-SIM-card slot. They each cost me $20. I was to discover that this was a real rip-off.

For calling the card was fine. But it turned out the card was not internet activated. At Timor Telecom this is a separate procedure/registration by the operator. So I went to the source.

The Timor Telecom office at landmark has good service, and after a couple of minutes they had provided me with an internet enabled SIM-card. Foreigners, like me, have to leave a copy of the passport to get registered. I had it with me and 15 minutes after entering the shop I wandered out with a functional micro-sim card. I was on line! And all it cost me wass $10. Perfect!

Topping up the phone with prepaid credits is an other matter. The information on the card is like this:

Following the procedure will get you nowhere. There is a secret code, and those without the secret knowledge will not be able to top up their credits. Having entered the code well over ten times still not succeeding I asked a young man to help me. NO problem he said, taking my phone and entering the number 100 in front of the code. Magic! It’s a horrible example of how “common” knowledge is forgotten, even by the service provider.

Timor Telecom provides stable and relatively fast internet via both EDGE and 3G. But they really need to redesign the information on their pulsa cards. And on their website as well. The secret procedure was not mentioned anywhere on their website.

Last time I had to manually enter the cellular data network setting. Seems like that procedure belongs to the past. But I will keep that info here for those of you settings problems:

My next challenge was to share my internet connection with a colleague. iPhone does that neatly, but to activate it you will need to enter information in the tethering apn interface. Navigate to this place by choosing Settings / General / Network / Cellular Data Network. Where it says Internet tethering you write internet like this:

This will get your tethering partner up and running following the normal iPhone tethering procedures. If you do not succeed at first with this procedure I suggest you try turning the Cellular data switch off and then on again. You will find it under Settings / General / Network / Cellular Data Network as well. It is my assumtion that this makes the phone reload the internet (apn) settings.

The secret code (100) you have to enter before you enter your pulsa code remains a secret for us inclined to read the instructons on the pulsa card.

But – Timor Telecom does the job better and better. 

A draft system was set up and documented in a former posting on this website. In the article, Environmental Spatial Data Infrastructure – technology, I described the system and some of the challenges. In this article I am taking it a bit further, hoping to stimulate to discussions about how such a system could be implemented.

This posting is about designing a clearinghouse predominantly intended for environmental data. It describes a work in progress. We are working on a requirements document and this posting is ment to inform interested parties about the work. Inputs to our work is both asked for and necessary.

What is a clearinghouse?

Clearinghouses come in many shapes and colors. Some are gigantic structures slowly crumbling under their own weight – being too complex and expensive to sustain use over time. Other systems are feather weight systems failing to do the job assigned to them. Other again are some are perfectly sized constructions which integrate well with other systems, flexible and varsatile. We are of course aiming at building the latter of these. There are no standards – and we’re starting from scratch on this one.

The objective of the clearinghouse is to provide in-depth information about the state and development of the environment. It shall present environmental topics from projects in a simple and easy-to-follow way, providing access to more detailed scientific presentations where such have been supplied by the contact organizations.

The following concept figure places a clearinghouse within in a context. As a concept figure it is not drawn with any particular country in mind.

Concept figure of a clearinghouse in an institutional and societal context.

Structure overview

Lets start with the main structure of our clearinghouse. Our partner has over the last three years completed two versions of a sensitivity atlas for a relatively big area. The next project is to establish a monitoring plan for the same area.

The structure for the monitoring program looks more or less like this:

Monitoring project structure

We have noted that the structure is general enough to be used with other projects. It is our ambition that the clearinghouse should be able to shoulder more than one project. If it does, it will probably also be used more, and one would avoid having a rather confused family of specialized sub-systems on the hosting organizations webpages. It would be easier for developers, partners, administrators and users.

Clearinghouse with the capacity to support many projects.

Now what if we also could be able to use this system with other partners? And what if others could use it as well? It depends on how well we design our system. Will it be general enough without rendering it useless for our purpose?

Our project context

We have taken great care to design the system as general as possible. Let’s have a look at the structure with more and different projects. See how the general structure (dark gray boxes) can be reflected in different projects in the figure below.

Clearinghouse with several projects structured around projects, themes and data groups.

The above figure at gives us an indication of how our data model fits with several of the prospective projects it shall serve.

Functionality

Within the overall structure we also have to consider which functionality the system should serve. The following represent a roundup of core functionalities of the system:

• Structured/hierarchical fact pages according to the presented projects. The pages will have text, tables, graphics and maps.
• The system facilitates input of data from stakeholders. The system administrator imports the data to the system.
• The system consists of both spatial and non-spatial data. In addition comes descriptive data (meta-data) providing the users with information necessary to know more about the provided data.
• The spatial data is provided to the user through maps. The maps consist of base layers (imagery to provide context) and the actual data. The actual data are “clickable” and will point to non-spatial data.
• A standard library of geographical objects is used. This means that the user will only have to upload references to the geographical objects and associated values.
• Non spatial data are files and online texts. The online texts provide context for files stored in the system. It gives the user an option to move through the system in a systematic way through hierarchies.
• Interactive request and comments box-which helps to provide feedback from stakeholders.
• Map layers are available through a content management system, as well as through OGC standards like Web Map Services (WMS), WFS and other.

It’s a tall order, but it can be done.

A challenge with the functionalities is that it is evident that the system will have two parallel data-supporting structures. One for the spatial data and one for the non-spatial data.

We’re thinking of keeping the spatial data in a separate database. The data is formated and made available as embeddable maps or tables giving an overview of the spatial data.

The non-spatial data provides a framework around the data. This means that we are using a content management system to establish a parallell structure to our custom made database. From within the content management system we then call for relevant tabular presentations and maps. It is not as neat as one could wish for, but drawing on the resources of a professional content management system is far better than building your own.

Our main work will therefore be in building components which will provide the content management with “consumables”.

Spatial data

Including maps to a clearinghouse adds to the complexity. There are two ways of dealing with the spatially related user data:

• The user uploads the geographical objects with attributes
• The user interact with predefined geographical objects

Uploading geographical objects would mean uploading shapefiles. Unfortunately shapefile uploading without a good quality assurance process could lead to several problems. These are some of them:

• Messed up or missing projections
• Corrupted files
• Inconsistent naming of files, objects and column names
• Duplicate objects
• Objects covering the same area but with different origins and quality
• Questionable legal status on the geographical objects
• It might be necessary to establish a user-role model within the admin module

We think the best thing in this case is to let one administrator handle the geographical objects. He or she should discuss with the users/partners what geographical objects are necessary and a proper process should then lead to the correct objects being imported.

The geographical objects in the system could be points, lines, polygons and multi-polygons. Basically whatever you can throw into a PostGIS table and attach a value to. An example could be waterbodies, districts, rivers, measuring points or  even Quarter Degree Grid Cells. The constraint is that the table of geographical objects should have a unique reference known to all potential providers of tabular data to be connected with the geographical object.

How will this pan out? We start with keeping quality assured geographical objects in the database. Next thing is to let users provide files with values and references to the geographical objects. Excel files would probably be easiest. Using SQL views within geoserver will make it easy for us to pull out the correct maps. Using SQL parameters could make it possible for us to limit the number of layer definitions in Geoserver.

In the end this will give us a map pretty much like this:

Districts drawn according to a value assigned to them. The data are random numbers from 0 to 6.

The users need to know which geographical objects are available, and which unique reference they answer to. The system should have a report engine able to provide them with a default list.

The picture is of course somewhat more complex. We have to restrict the value sets the available for the users as well. This is because we most probably will have to keep a limited number of styles available for the users. Many styles would add to the complexity.

By restricting the data contributors we are hopefully helping the users of the system getting more readable maps.

Styling the maps

The map styling is handled by keeping a default styling in the database. We are considering three styling standards. We will have to consider more depending on how many layers we will have in one map.

Styling sketches. How should we handle the needs and wants of styling?

We might also be able to implement the new styling transformation functions as documented in Geoserver 2.2 and this might ease our workload somewhat.

Data model

A data model has been sketched using MS Access. The clearinghouse will however be running on top of a PostGIS database. The database will not integrate directly with the one supporting the content management system.

As one can see from the model the hierarchy is supported by the following tables:

• tbl_h_project
• tbl_h_theme
• tbl_h_datagroup

Draft database model designed using MS Access. The final database will be safely set up in PostGIS.

We will leave it to the interested reader to look in more detail on the data model.

Pulling it all together

As explained earlier the system will rely on many modules, some of which are custom made and some of which are standard systems. Luckily we have now come to where we have many good standard systems many of which are made by OpenGeo.

The following are the standard modules and tools which will be used:

Standard modules to be used in the clearinghouse.

The main effort for the clearinghouse project will lie in the custom made modules:

Custom made modules for the clearinghouse system.

The figure below indicates the relation between the different modules. The custom made modules are in light gray and will have to be specified and developed.

Shows relation between data input/output (white boxes), custom modules (light gray) and standard modules (dark gray).

The custom modules will leave a lot of work on the structure to be done within WordPress. This means the administratir will have to maintain a mirror structure of the one in the clearinghouse database for non spatial data. A properly designed lists module will be helpfull.

Challenges and questions

Challenges to a system like this are likely to be many. Here are some of them:

• How do you handle the many styles necessary to keep several layers in one map?
  • The styling standard used in geoserver (SLD) is flexible. We will probably set up some standard styles and also keep a couple of dynamic styles using the interpolate functionality in SLD.
• Why did we choose OpenGeoSuite over GeoNode?
  • GeoNode is an excellent tool for management and presentation of spatial data. But currently the project stops there. In a clearinghouse it is necessary to integrate many pieces of information – both spatial and non-spatial. The data model and other tools to integrate the information is bluntly speaking too complex for Geonode. That is not to say that Geonode cannot be a tool used to play around with the data we present. It can be – and it probably also will be. But at this stage not by us.
• How do we deal with ownership?
  • Using open source modules it makes sense not to break it. In an initial phase, up until the first version is ready we will work our way forward together with our developers. The custom made modules will then be released into the wild on a suitable  collaborative platform for open source coding.
• What about meta-data?
  • Meta data is an important part of web based services. We were at some point contemplating including GeoNetwork into the system. At this point we will have to focus on preparing a primary level functional clearinghouse. Geonetwork could be added later.

Concluding remarks

This article describes my initial thoughts on such a system. We are working on a specification of the system and we do have funding for some development. We expect to have a draft specifications document ready for distribution sometime in September. We expect our developer has a beta ready by the end of the year.

At this point we are looking for feedback on the above text. Are there issues we should take care in covering better? Does similar systems exist? Where should we host the publicly available code?