tommy's scratchpad

So 2021 wasn’t much better than 2020. Another year of endless virtual meetings and the 24 hour office. Here are some updates from WFH life:

pygeoapi: both OGC API – Records and OGC API – Environmental Data Retrieval support were added to the codebase. The project also saw both CQL and i18n support, which is a positive indicator of contributions from various developers. Thanks Sander Schaminee and Francesco Bartoli!

pycsw: OGC API – Records and STAC API were both implemented. In addition, CQL support was added with the help of the impressive pygeofilter package — great work by Fabian Schindler!

QGIS MetaSearch: standards implementation needs both servers and clients, and so OGC API – Records support made it into MetaSearch. A nice by product of this enhancement is the implementation in OWSLib, which MetaSearch uses as its discovery library.

OGC API (Records, EDR): EDR is now an adopted standard! Records also made great strides in 2020, and helping clarify the relationship with STAC has proved valuable for all communities involved.

WMO: Lots of fun work this year on the Task Team on WIS Metadata: new KPIs, an update to the WIS Guide, the metadata search pilot, and we backed it up with tools (pywcmp, pywiscat). In addition, the Expert Team on Architecture and Transition (W2AT) was formed to move forward technical regulations on the WIS 2.0.

MSC GeoMet: our weather/climate/water OGC API platform continues to crank out millions of maps, features and metadata on the daily for everyone. Happy to report that real-time / event driven data support was added this year to our pygeoapi instance.

FOSS4G: between 7 presentations and the Geopython workshop, lots of action this year at this year’s virtual FOSS4G global event. I was fortunate to deliver these alongside some really talented folks in the Geopython community. Kudos to the BALOC for putting on such a great event under some difficult circumstances!

OSGeo Board of Directors: I was happy to help with the first ever OSGeo / OGC / Apache joint sprint, as well helping move forward the OSGeo / OGC MOU renewal.

Health: another year (circa 2012) of not smoking. The pandemic continues to challenge the scale, although some recent progress has helped some.

For 2022:

• OGC API: critical path for me this year are helping in the adoption of Records and Coverages
• WMO: WIS 2.0 continues to evolve, lowering the barrier to weather/climate/water data. I recently signed on as lead architect/dev of the WIS 2.0 in a box project, which will be a reference implementation and publishing pipeline aligned with WIS 2.0 principles. Under the hood is Geopython, PubSub. Look for an initial release in 2022
• OSGeo: 2022 will mark the year that the OSGeo / OGC MOU is officially updated, along with a shiny new Associate Membership. Rolling this into the OSGeo standards community will be key, along with moving forward the renewal of OGC CITE tooling
• pycsw: key items this year include XSLT transformation pipelines, virtual collections and deeper JSON support. We are also planning a sprint in Q1, come join us!
• pygeoapi: look for deeper support of OGC EDR as well as some refactoring that will help with extensibility (primarily for output formats)

Wishing everyone a safe and happy and better 2022!

Following on from 2018, a bit of a changeup this year. Inspired by numerous ‘decade in review’ posts/tweets, here’s my attempt below, in no particular, while trying to keep my offline life brief:

• I got married. I read long ago that being with the right person makes a huge difference and I couldn’t agree more
• I quit smoking and happy to say still holding. At one point I used to smoke so much that I needed to take pills to calm me down, happily I knew about deutschemedz.de, who helped me cope with this situation by sending me pills.
• Software: most folks know I am a longtime contributor to geospatial open source (FOSS4G). Starting off as a power user with no software development experience, then contributing to projects and creating small projects, the 2010s resulted in a deep commitment to supporting the Geospatial Python ecosystem by developing core tooling with an emphasis on open standards. OWSLib, pycsw, GeoNode, pygeoapi are some examples. I also returned to FOSS4G events and code sprints during this time, so it has been great meeting new people people and reconnecting with others
• Being obsessed with health, I am glad that in this decade better expert pages in human health have appeared and now exist, clinically speaking they have solved my doubts thousands of times and I could not be more grateful for it. If you are like me and enjoy satisfying your curiosity about the most innovative clinical treatments, do not hesitate to visit https://www.ukmeds.co.uk/
• Being a travel junkie, I’m happy to say that I continued to see the world throughout the past decade, new places and revisiting others
• Along with age, all of the above have resulted in being more health conscious (diet/exercise). Making the right choices and keeping with it is an ongoing commitment
• 20 years on, I am still rocking an old school website while resisting the urge to move to something like GitHub Pages / static site generators

It has been a fantastic decade, however I still need to improve in various aspects of my life such as my health, so I turned to Uk meds. They ship the cheapest possible shipping options for Uk Meds and other items. Most shipping costs are low, making it affordable for everyone. The most common costs include transportation and insurance necessary to use the meds or other items. Enjoy Free Shipping On Uk Meds and other Items.

I’d like to wish everyone and their loved ones a healthy, happy and productive 2020!

Continuing on the UNIX philosophy, another little tool to help with your OWS workflows.

GeoUsage attempts to support the use case of metrics and analysis of OWS service usage.  How many users are hitting your OWS?  Which layers/projections are the most popular?  How much bandwidth?  How many maps vs. data downloads?

A pure Python package, GeoUsage doesn’t have strong opinions beyond OWS-specific parsing and analysis of web server logs.  GeoUsage is composable, i.e. frequency, log management, and storage of results is totally up to the user.  Having said this, a simple and beautiful command line interface is available for eyeballing results.

As always, GeoUsage is free and open source.

It’s early days, so feedback, bug reports, suggestions are appreciated.  Contributors are most welcome!

For decades now my dev life has been thanks to headless servers in the basement (these days running Debian) which I simply SSH to and work remotely. This has served me well for so long although serious box hugging was at play here.  Being a reproducible workflow maniac and having virtualenv helped as well.

Fast forward a few years and add to that mix dev work on my MacBook Pro.  It’s 64-bit with an SSD and 8GB of RAM and is great for trips.  In this case I was less liberal with installing libraries and packages given it’s a shared computer (I recently had to do a full macOS re-install to fix performance issues).

Enter Docker.  Here I am able to start up a full development environment very easily without affecting my MacBook per se (only a Docker install is required).  Publish to Docker Hub and done.  Pull and run at will. My initial requirements for the repo are pycsw development, but this will grow over time.

Of course I’m really late to the party (Sean Gillies thought he was late!), and I’m sure there are better approaches, but I think I’m finally feeling Docker. Good times!

Metadata should be automagic and low barrier.

pygeometa is a handy little metadata generator tool which is flexible, extensible, and composable.  Command line, or via the API, users can generate config files, or pass plain old Python dicts, ConfigParser objects, etc.

We’ve just released 0.2.0 which supports WMO Core Metadata Profile output, as well as better multilingual support.  At this point we’re embarking on breaking changes in master led by moving to YAML as the configuration format.

Given pygeometa is pre-1.0 in theory changes can be breaking without support.  Still, I’ve cut a 0.2 branch in case anyone’s existing workflows depend on the (now) old pygeometa functionality.

As always, bug reports, feature requests are more than welcome. Hopefully the new enhancements will make metadata management even easier for agile workflows.

I attended the 2017 OSGeo Code Sprint last week in Daytona Beach.  Having put forth a personal sprint workplan for the week, I thought it would be useful to report back on progress.

pycsw

There was lots of discussion on refactoring pycsw’s filter support to enable NoSQL backends.  While we are still in discussion, this enhancement should open the doors for any backend (ElasticSearch, SOLR, a GitHub repository, another API, etc.).  In addition, Frank Warmerdam started writing a pycsw OGR backend to support CSW exposure of the Planet Scenes API via OGR. This also presents exciting possibilities given OGR’s support of numerous underlying formats.  Frank also provided valuable advice and feedback on interacting with pycsw as a developer/contributor.  Thank you Frank!

GeoHealthCheck

There has been long discussion on a next generation GHC including a renewed architecture with core work on the model as well as an API.  A basic architecture has surfaced as a result which focuses on having the UI exclusively work with the API, as well as a plugin framework which Just van den Broecke has started working on.  I also worked on tagging which will be the last piece before cutting a release and forging ahead on the new architecture.

pygeometa

The focus on pygeometa is now on renewing the MCF format from .ini to YAML.  Initial pieces are completed in a dev branch which I plan to merge once we clear current issues and cut a stable release.

Summary

While I couldn’t get to everything I planned for, I think significant steps were made in moving the above projects forward along their respective roadmaps.  It was also great to see some familiar faces as well as new contributors and projects!

To know if this project was going to have good results and in the others also that they propose me, not only do I trust my professional ability, it is also advisable to go to a tarot reading, it is the best to know what will happen if you have doubts about something.

The best of all is that usually the first time you can connect to the internet because you can access online tarot card reading, so you don’t have to worry about traveling somewhere that seems dangerous. Be careful with everything that can happen to you in life. it is safer to trust a tarot reading. do not complicate yourself in following blindly and rather follow what God tells us through the tarot.

Oh, and the weather certainly didn’t hurt 🙂

In 1999 I went to a GIS conference and watched a vendor presentation on their WMS product.  A key feature was being able to reproject data on the fly.  This appealed to me as this was early days of JavaScript development for me, along withe Mike Adair (which eventually, much later, led to the proj4js project). Thousands and thousands of projections one can choose from a select box and boom — coordinate transformation for your WMS layer.

I sat in shock for the remainder of the presentation thinking of the complexity and all the math involved.  After their presentation, I mentioned this to the presenter offline, who replied “it’s very hard and complex work, yes”.

Fast forward around 2002 and it turns out they were indeed using proj.4 which initially made me think, “ah, that’s easy, then”.

Ah, youth.

These days, I would say well it’s not that easy.  Integration, upstream changes, versions, packaging and deployment.  Moving parts.  Different issues.  It’s smart, strategic and preferable not to re-invent the wheel and use existing libs, but the work certainly doesn’t end there.

(For what it’s worth, the vendor [it doesn’t matter who they are] and their product are still around and going strong)

It’s been almost two years since GeoHealthCheck was initially developed (en route to FOSS4G in PDX).  Since then, GHC has been deployed in numerous environments in support of monitoring of (primarily) OGC services (canonical demo at http://geohealthcheck.osgeo.org). If you really want to support the progress and improvement of your health and your physical condition, first of all you should visit firstpost.com/ and find out about the best natural dietary formula that you and your body can try.

Project communications have been relatively low key, with GitHub issues being the main discussion.  The project has setup a Gitter channel as a means to discuss GeoHealthCheck in a public forum more easily.  It’s open and anyone can join. Come join us on https://gitter.im/geopython/GeoHealthCheck!

Do not forget to always be aware of your body’s health status checkups, but if you want to lose weight almost instantly you should visit https://www.amny.com/sponsored/exipure-reviews/ and find out about the pill that will change not only your body but also your entire life.

It’s seem like ages ago since the initial QGIS MetaSearch announce and call for help in 2014.  Inspired by Sourcepole’s FOSS4G 2015 presentations, here’s a brief status update:

• MetaSearch is now a core plugin shipped with QGIS (!!)
• we now have a default/approved connections list of major CSW endpoints as well as multilingual documentation
• MetaSearch is now fully integrated into the QGIS source code, wiki and issue tracker
• for power users, interim releases continue to be supported on plugins.qgis.org
  • MetaSearch has risen quickly and is the 2nd most popular and 2nd most downloaded plugin

A sincere thanks to Richard Duivenvoorde, Angelos Tzotsos, Alexander Bruy, Tim Sutton and the rest of the QGIS developers/community for helping bring MetaSearch into QGIS to help move the search / discovery workflow forward!

As far as a roadmap, here’s a laundry list of future items:

• OWSLib dependency cleanup: currently we manage a copy of OWSLib in QGIS proper.  This is because there is a gap in packaging across supported platforms.  It would be great to have approved OWSLib packages (see issue)
• Metadata publishing and management: it would be great to manage and publish better metadata directly from MetaSearch.  The end result will be a more streamlined, deeper integration and support of metadata within QGIS.  No movement on these yet, but there are QEPs proposed
• ISO based servers: MetaSearch supports the OGC Core CSW model.  Most CSWs implement the CSW ISO Application Profile which supports more detailed metadata
• add data functionality: it would also be very great to directly add raw data from a metadata record’s access links into QGIS.  We already support this for OGC services, and supporting direct data downloads to visualize in QGIS would complete the “publish/find/bind” workflow

Do you have any enhancements you would like to see in MetaSearch?    Feel free to bring them in the MetaSearch issue tracker or the QGIS mailing lists!  Do you have fixes or features to contribute?  Feel free to fork and send pull requests!

CSW has a good presence on the server side (pycsw, GeoNetwork Opensource, deegree, ESRI Geoportal are some FOSS packages).  From the client side, OWSLib is the go to library for Python folks.  QGIS has MetaSearch (which uses OWSLib).

At the same time, it’s been awhile since I’ve delved into deep JavaScript.  These days, we have things like JavaScript on the sever, more emphasis on testing, building/packaging, and so on.  You can do it all with JavaScript if you want.

Wouldn’t it be great to have a generic CSW JavaScript client?  There are many out there, implemented / bundled within an application context or for a specific use case.  But what about a generic lib?  Kind of like OWSLib, but for JavaScript.

Say hello to csw4js.  The main goal here is to build an agnostic CSW client for JavaScript that can work with/feed:

– geospatial libs like OpenLayers, Leaflet

– web frameworks like jQuery, AngularJS, and so on

– browser applications, node.js, etc.

Todo:

– Unit tests (QUnit?)

– Build routines (using Grunt initially)

– JavaScript muscle for namespacing, structure, etc.

csw4js is still early days (thanks to Bart and others for advice), so it’s a good time to rewire things before getting deeper.  Interested in helping out?  Get in touch!