GeoUsage: Log Analyzer for OGC Web Services

Continuing on the UNIX philosophy, another little tool to help with your OWS workflows.

GeoUsage attempts to support the use case of metrics and analysis of OWS service usage.  How many users are hitting your OWS?  Which layers/projections are the most popular?  How much bandwidth?  How many maps vs. data downloads?

A pure Python package, GeoUsage doesn’t have strong opinions beyond OWS-specific parsing and analysis of web server logs.  GeoUsage is composable, i.e. frequency, log management, and storage of results is totally up to the user.  Having said this, a simple and beautiful command line interface is available for eyeballing results.

As always, GeoUsage is free and open source.

It’s early days, so feedback, bug reports, suggestions are appreciated.  Contributors are most welcome!

Cheers to 2017

Here we go again! Following on from last year, a summary of my 2017:

– pycsw: the lightweight CSW server continues provide stable, composable, and compliant CSW services.  Highlights include:

  • an official code of conduct
  • Docker image
  • testing framework enhancements
  • code coverage support
  • custom repository plugin filter parsing

– MapServer metadata: at long last RFC82 has been implemented in master and will be available in 7.2.  This enhancement will bring more fulsome layer/feature type/coverage metadata support to OGC web services in MapServer

– QGIS metadata: at long last metadata support is building in QGIS!  There is now an official internal schema as well as work in master around the editor

– parsers: in the back to basics category, pyshadoz and pynumeric were built as small, composable Python packages to feed operational data processing workflows.  At the end of the day (or the beginning depending on how you look at it), it’s all about the data, and data formats are everything

– other geopython:

  • pygeometa: another little toolset for discovery metadata, YAML is now the supported configuration format
  • OWSLib: continues its (still pre-1.0!) journey, making OWS suck less
  • GeoHealthCheck: Just has been leading the charge on improvements and fixes to GHC (thanks Just!)

– health:

  • another year (circa 2012) of not smoking
  • I’m happy to continue commitment to the Greek/Mediterranean diet.  Another 10lbs would be great, however I’m very happy with my overall state of weight management (first time since starting a desk job in 1997).  Some small adjustments and minor ups/downs, but overall stable and continued following of olivetomato.com
  • I’ve also given up coffee after 25 years and would highly recommend it!

For 2018:

-pycsw: the pycsw team is attending the OSGeo Code Sprint 2018 in Bonn, and will be working on getting 2.2 out the door as well as pycsw 3.0 planning (more info on this soon)

– QGIS metadata: looking forward to continuing this effort with enhancements to MetaSearch to allow for CSW publishing

– Pydap: I’m hoping to make some serious movement in deployment Pydap with some dayjob projects.  Key focus is hardening deployment as well as stability

Golang: some may have seen some recent work on geocatalogo.  Given the nature of Go and simplicity of deployment, there is opportunity in the geospatial community for Golang packages in providing geospatial discovery and access services which are based on focused use cases / workflows, are dead simple to install, deploy and scale, and blazing fast.  While these packages may not be as full-featured or mature as current OSGeo projects, there’s certainly room for them and hopefully we can see a Golang geospatial community evolve over the next few years

– PyWPS OSGeo incubation: got a bit closer but didn’t finish this off in 2017.  A sprint will close this one and send for graduation

– OWSLib tests: could this be the year that OWSLib unit tests are finally not always failing!?

In summary, another year of firm belief in the Unix philosophy.

I’d like to wish everyone and their loved ones a healthy, happy and productive 2018!

Hello Docker

For decades now my dev life has been thanks to headless servers in the basement (these days running Debian) which I simply SSH to and work remotely. This has served me well for so long although serious box hugging was at play here.  Being a reproducible workflow maniac and having virtualenv helped as well.

Fast forward a few years and add to that mix dev work on my MacBook Pro.  It’s 64-bit with an SSD and 8GB of RAM and is great for trips.  In this case I was less liberal with installing libraries and packages given it’s a shared computer (I recently had to do a full macOS re-install to fix performance issues).

Enter Docker.  Here I am able to start up a full development environment very easily without affecting my MacBook per se (only a Docker install is required).  Publish to Docker Hub and done.  Pull and run at will. My initial requirements for the repo are pycsw development, but this will grow over time.

Of course I’m really late to the party (Sean Gillies thought he was late!), and I’m sure there are better approaches, but I think I’m finally feeling Docker. Good times!

pygeometa: new release, hello YAML

Metadata should be automagic and low barrier.

pygeometa is a handy little metadata generator tool which is flexible, extensible, and composable.  Command line, or via the API, users can generate config files, or pass plain old Python dicts, ConfigParser objects, etc.

We’ve just released 0.2.0 which supports WMO Core Metadata Profile output, as well as better multilingual support.  At this point we’re embarking on breaking changes in master led by moving to YAML as the configuration format.

Given pygeometa is pre-1.0 in theory changes can be breaking without support.  Still, I’ve cut a 0.2 branch in case anyone’s existing workflows depend on the (now) old pygeometa functionality.

As always, bug reports, feature requests are more than welcome. Hopefully the new enhancements will make metadata management even easier for agile workflows.

OSGeo Daytona Beach Code Sprint 2017 redux

I attended the 2017 OSGeo Code Sprint last week in Daytona Beach.  Having put forth a personal sprint workplan for the week, I thought it would be useful to report back on progress.

pycsw

There was lots of discussion on refactoring pycsw’s filter support to enable NoSQL backends.  While we are still in discussion, this enhancement should open the doors for any backend (ElasticSearch, SOLR, a GitHub repository, another API, etc.).  In addition, Frank Warmerdam started writing a pycsw OGR backend to support CSW exposure of the Planet Scenes API via OGR. This also presents exciting possibilities given OGR’s support of numerous underlying formats.  Frank also provided valuable advice and feedback on interacting with pycsw as a developer/contributor.  Thank you Frank!

GeoHealthCheck

There has been long discussion on a next generation GHC including a renewed architecture with core work on the model as well as an API.  A basic architecture has surfaced as a result which focuses on having the UI exclusively work with the API, as well as a plugin framework which Just van den Broecke has started working on.  I also worked on tagging which will be the last piece before cutting a release and forging ahead on the new architecture.

pygeometa

The focus on pygeometa is now on renewing the MCF format from .ini to YAML.  Initial pieces are completed in a dev branch which I plan to merge once we clear current issues and cut a stable release.

Summary

While I couldn’t get to everything I planned for, I think significant steps were made in moving the above projects forward along their respective roadmaps.  It was also great to see some familiar faces as well as new contributors and projects!

Oh, and the weather certainly didn’t hurt 🙂

Cheers to 2016

It’s been quite awhile since I did one of these, so here goes.  Some notables from 2016:

  • pycsw: the release of 2.0 “Doug” provided the first OGC compliant CSW 3.0 implementation, as well as Python 3 support.  These two major enhancements provide the long term backbone for the project moving into the future
  • GeoHealthCheck: GHC provided the inspiration for the Harvard Hypermap project.  In addition, the project is being used in numerous internal environments and has caught the itch of Just van den Broecke! It’s amazing what happens when you put a UI on top of workflows
  • PyWPS: version 4.0 was released which represented a major update/rewrite/licence change of the project.  For WOUDC, we’ve implemented PyWPS as part of real-time workflows for data validation.  Finally, the project has moved along the OSGeo incubation process nicely and is hours away from being submitted for project graduation
  • pygeometa: the little metadata creation tool now supports the WMO Core Metadata Profile
  • GeoNode: now an OSGeo project!
  • health
    • another year (circa 2012) of not smoking
    • I lost 35 lbs in 2016 thanks to a true, deep commitment to the Greek/Mediterranean diet. A huge thank you goes out to Olive Tomato, which has provided awesome recipes and advice

For 2017:

  • pycsw: look for some big improvements to our test suite, as well as ElasticSearch support
  • pygeometa: move to YAML as the configuration format
  • PyWPS OSGeo incubation: we’re almost there! Hoping to complete this by spring
  • GeoHealthCheck: implementing a GHC API and plugin mechanism are two key enhancements which we will hopefully tackle at the OSGeo Code Sprint in Daytona Beach.  As well, as following the developments of newly formed OGC Quality of Service and Experience Domain Working Group

Wishing you and yours a healthy and happy 2017!

Software is Hard: Through the Years

In 1999 I went to a GIS conference and watched a vendor presentation on their WMS product.  A key feature was being able to reproject data on the fly.  This appealed to me as this was early days of JavaScript development for me, along withe Mike Adair (which eventually, much later, led to the proj4js project). Thousands and thousands of projections one can choose from a select box and boom — coordinate transformation for your WMS layer.

I sat in shock for the remainder of the presentation thinking of the complexity and all the math involved.  After their presentation, I mentioned this to the presenter offline, who replied “it’s very hard and complex work, yes”.

Fast forward around 2002 and it turns out they were indeed using proj.4 which initially made me think, “ah, that’s easy, then”.

Ah, youth.

These days, I would say well it’s not that easy.  Integration, upstream changes, versions, packaging and deployment.  Moving parts.  Different issues.  It’s smart, strategic and preferable not to re-invent the wheel and use existing libs, but the work certainly doesn’t end there.

(For what it’s worth, the vendor [it doesn’t matter who they are] and their product are still around and going strong)

GeoHealthCheck support on Gitter

It’s been almost two years since GeoHealthCheck was initially developed (en route to FOSS4G in PDX).  Since then, GHC has been deployed in numerous environments in support of monitoring of (primarily) OGC services (canonical demo at http://geohealthcheck.osgeo.org).

Project communications have been relatively low key, with GitHub issues being the main discussion.  The project has setup a Gitter channel as a means to discuss GeoHealthCheck in a public forum more easily.  It’s open and anyone can join.

Come join us on https://gitter.im/geopython/GeoHealthCheck!

QGIS MetaSearch status and update

It’s seem like ages ago since the initial QGIS MetaSearch announce and call for help in 2014.  Inspired by Sourcepole’s FOSS4G 2015 presentations, here’s a brief status update:

A sincere thanks to Richard Duivenvoorde, Angelos Tzotsos, Alexander Bruy, Tim Sutton and the rest of the QGIS developers/community for helping bring MetaSearch into QGIS to help move the search / discovery workflow forward!

As far as a roadmap, here’s a laundry list of future items:

  • OWSLib dependency cleanup: currently we manage a copy of OWSLib in QGIS proper.  This is because there is a gap in packaging across supported platforms.  It would be great to have approved OWSLib packages (see issue)
  • Metadata publishing and management: it would be great to manage and publish better metadata directly from MetaSearch.  The end result will be a more streamlined, deeper integration and support of metadata within QGIS.  No movement on these yet, but there are QEPs proposed
  • ISO based servers: MetaSearch supports the OGC Core CSW model.  Most CSWs implement the CSW ISO Application Profile which supports more detailed metadata
  • add data functionality: it would also be very great to directly add raw data from a metadata record’s access links into QGIS.  We already support this for OGC services, and supporting direct data downloads to visualize in QGIS would complete the “publish/find/bind” workflow

Do you have any enhancements you would like to see in MetaSearch?    Feel free to bring them in the MetaSearch issue tracker or the QGIS mailing lists!  Do you have fixes or features to contribute?  Feel free to fork and send pull requests!

CSW Client Library for JavaScript: the Adventure Begins

CSW has a good presence on the server side (pycsw, GeoNetwork Opensource, deegree, ESRI Geoportal are some FOSS packages).  From the client side, OWSLib is the go to library for Python folks.  QGIS has MetaSearch (which uses OWSLib).

At the same time, it’s been awhile since I’ve delved into deep JavaScript.  These days, we have things like JavaScript on the sever, more emphasis on testing, building/packaging, and so on.  You can do it all with JavaScript if you want.

Wouldn’t it be great to have a generic CSW JavaScript client?  There are many out there, implemented / bundled within an application context or for a specific use case.  But what about a generic lib?  Kind of like OWSLib, but for JavaScript.

Say hello to csw4js.  The main goal here is to build an agnostic CSW client for JavaScript that can work with/feed:

– geospatial libs like OpenLayers, Leaflet

– web frameworks like jQuery, AngularJS, and so on

– browser applications, node.js, etc.

Todo:

– Unit tests (QUnit?)

– Build routines (using Grunt initially)

– JavaScript muscle for namespacing, structure, etc.

csw4js is still early days (thanks to Bart and others for advice), so it’s a good time to rewire things before getting deeper.  Interested in helping out?  Get in touch!

 

Modified: 28 April 2014 18:00:00 EST