Multiple changes in the API, schemas and UI

Check out the CHANGELOG.md file for more information

.gitlab-ci.yml

@@ -28,6 +28,7 @@ test-3.5:
 
 test-2.7:
   <<: *test_definition
+  allow_failure: true
   variables:
     PYTHON_VERSION: "2.7"
 

.makefiles/python.mk

@@ -29,7 +29,7 @@ build: $(addprefix build-, $(PYVERSIONS)) ## Build all images / python versions
 	docker tag $(IMAGEWTAG)-python$(PYMAIN) $(IMAGEWTAG)
 
 build-%: version Dockerfile-%  ## Build a specific version (e.g. build-2.7)
-	docker build -t '$(IMAGEWTAG)-python$*' -f Dockerfile-$* .;
+	docker build --pull -t '$(IMAGEWTAG)-python$*' -f Dockerfile-$* .;
 
 dev-%: ## Launch a specific development environment using docker (e.g. dev-2.7)
 	@docker start $(NAME)-dev$* || (\

CHANGELOG.md

+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+* Objects can control the keys that will be used in `serialize`/`jsonld`/`as_dict` by specifying a list of keys in `terse_keys`.
+e.g.
+```python
+>>> class MyModel(senpy.models.BaseModel):
+...   _terse_keys = ['visible']
+...   invisible = 5
+...   visible = 1
+...
+>>> m = MyModel(id='testing')
+>>> m.jsonld()
+{'invisible': 5, 'visible': 1, '@id': 'testing'}
+>>> m.jsonld(verbose=False)
+{'visible': 1}
+```
+* Configurable logging format.
+* Added default terse keys for the most common classes (entry, sentiment, emotion...).
+* Flag parameters (boolean) are set to true even when no value is added (e.g. `&verbose` is the same as `&verbose=true`).
+* Plugin and parameter descriptions are now formatted with (showdown)[https://github.com/showdownjs/showdown].
+* The web UI requests extra_parameters from the server. This is useful for pipelines. See #52
+* First batch of semantic tests (using SPARQL)
+
+### Changed
+* `install_deps` now checks what requirements are already met before installing with pip.
+* Help is now provided verbosely by default
+* Other outputs are terse by default. This means some properties are now hidden unless verbose is set.
+* `sentiments` and `emotions` are now `marl:hasOpinion` and `onyx:hasEmotionSet`, respectively.
+* Nicer logging format
+* Context aliases (e.g. `sentiments` and `emotions` properties) have been replaced with the original properties (e.g. `marl:hasOpinion` and `onyx:hasEmotionSet**), to use aliases, pass the `aliases** parameter.
+* Several UI improvements
+  * Dedicated tab to show the list of plugins
+  * URLs in plugin descriptions are shown as links
+  * The format of the response is selected by clicking on a tab instead of selecting from a drop-down
+  * list of examples
+  * Bootstrap v4
+* RandEmotion and RandSentiment are no longer included in the base set of plugins
+* The `--plugin-folder` option can be used more than once, and every folder will be added to the app.
+
+### Deprecated
+### Removed
+* Python 2.7 is no longer test or officially supported
+### Fixed
+* Plugin descriptions are now dedented when they are extracted from the docstring.
+### Security
+

Makefile

@@ -5,7 +5,7 @@ IMAGENAME=gsiupm/senpy
 
 # The first version is the main one (used for quick builds)
 # See .makefiles/python.mk for more info
-PYVERSIONS=3.5 2.7
+PYVERSIONS=3.6 3.7
 
 DEVPORT=5000
 

Procfile

-web: python -m senpy --host 0.0.0.0 --port $PORT --default-plugins
+web: python -m senpy --host 0.0.0.0 --port $PORT

README.rst

 .. image:: img/header.png
    :width: 100%
-   :target: http://demos.gsi.dit.upm.es/senpy
+   :target: http://senpy.gsi.upm.es
 
 .. image:: https://travis-ci.org/gsi-upm/senpy.svg?branch=master
    :target: https://travis-ci.org/gsi-upm/senpy
 
+.. image:: https://lab.gsi.upm.es/senpy/senpy/badges/master/pipeline.svg
+   :target: https://lab.gsi.upm.es/senpy/senpy/commits/master
+
+.. image:: https://lab.gsi.upm.es/senpy/senpy/badges/master/coverage.svg
+   :target: https://lab.gsi.upm.es/senpy/senpy/commits/master
+
+.. image:: https://img.shields.io/pypi/l/requests.svg
+   :target: https://lab.gsi.upm.es/senpy/senpy/
+
 Senpy lets you create sentiment analysis web services easily, fast and using a well known API.
 As a bonus, senpy services use semantic vocabularies (e.g. `NIF <http://persistence.uni-leipzig.org/nlp2rdf/>`_, `Marl <http://www.gsi.dit.upm.es/ontologies/marl>`_, `Onyx <http://www.gsi.dit.upm.es/ontologies/onyx>`_) and formats (turtle, JSON-LD, xml-rdf).
 
@@ -12,7 +21,7 @@ Have you ever wanted to turn your sentiment analysis algorithms into a service?
 With senpy, now you can.
 It provides all the tools so you just have to worry about improving your algorithms:
 
-`See it in action. <http://senpy.cluster.gsi.dit.upm.es/>`_
+`See it in action. <http://senpy.gsi.upm.es/>`_
 
 Installation
 ------------
@@ -38,9 +47,9 @@ If you want to install senpy globally, use sudo instead of the ``--user`` flag.
 
 Docker Image
 ************
-Build the image or use the pre-built one: ``docker run -ti -p 5000:5000 gsiupm/senpy --default-plugins``.
+Build the image or use the pre-built one: ``docker run -ti -p 5000:5000 gsiupm/senpy``.
 
-To add custom plugins, add a volume and tell senpy where to find the plugins: ``docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins gsiupm/senpy --default-plugins -f /plugins``
+To add custom plugins, add a volume and tell senpy where to find the plugins: ``docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins gsiupm/senpy -f /plugins``
 
 
 Developing
@@ -125,6 +134,16 @@ For more information, check out the `documentation <http://senpy.readthedocs.org
 ------------------------------------------------------------------------------------
 
 
+Python 2.x compatibility
+------------------------
+
+Keeping compatibility between python 2.7 and 3.x is not always easy, especially for a framework that deals both with text and web requests.
+Hence, starting February 2019, this project will no longer make efforts to support python 2.7, which will reach its end of life in 2020.
+Most of the functionality should still work, and the compatibility shims will remain for now, but we cannot make any guarantees at this point.
+Instead, the maintainers will focus their efforts on keeping the codebase compatible across different Python 3.3+ versions, including upcoming ones.
+We apologize for the inconvenience.
+
+
 Acknowledgement
 ---------------
 This development has been partially funded by the European Union through the MixedEmotions Project (project number H2020 655632), as part of the `RIA ICT 15 Big data and Open Data Innovation and take-up` programme.

config.py

-import os
-
-SERVER_PORT = os.environ.get("SERVER_PORT", 5000)
-DEBUG = os.environ.get("DEBUG", True)

docs/Makefile

@@ -24,6 +24,7 @@ I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
 help:
 	@echo "Please use \`make <target>' where <target> is one of"
 	@echo "  html       to make standalone HTML files"
+	@echo "  entr       to watch for changes and continuously make HTML files"
 	@echo "  dirhtml    to make HTML files named index.html in directories"
 	@echo "  singlehtml to make a single large HTML file"
 	@echo "  pickle     to make pickle files"
@@ -49,6 +50,9 @@ help:
 clean:
 	rm -rf $(BUILDDIR)/*
 
+entr:
+	while true; do ag -g rst | entr -d make html; done 
+
 html:
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
 	@echo

docs/SenpyClientUse.ipynb

-%% Cell type:markdown id: tags:
-
-# Client
-
-%% Cell type:markdown id: tags:
-
-The built-in senpy client allows you to query any Senpy endpoint. We will illustrate how to use it with the public demo endpoint, and then show you how to spin up your own endpoint using docker.
-
-%% Cell type:markdown id: tags:
-
-Demo Endpoint
--------------
-
-%% Cell type:markdown id: tags:
-
-To start using senpy, simply create a new Client and point it to your endpoint. In this case, the latest version of Senpy at GSI.
-
-%% Cell type:code id: tags:
-
-``` python
-from senpy.client import Client
-
-c = Client('http://latest.senpy.cluster.gsi.dit.upm.es/api')
-```
-
-%% Cell type:markdown id: tags:
-
-Now, let's use that client analyse some queries:
-
-%% Cell type:code id: tags:
-
-``` python
-r = c.analyse('I like sugar!!', algorithm='sentiment140')
-r
-```
-
-%% Cell type:markdown id: tags:
-
-As you can see, that gave us the full JSON result. A more concise way to print it would be:
-
-%% Cell type:code id: tags:
-
-``` python
-for entry in r.entries:
-      print('{} -> {}'.format(entry['text'], entry['sentiments'][0]['marl:hasPolarity']))
-```
-
-%% Cell type:markdown id: tags:
-
-We can also obtain a list of available plugins with the client:
-
-%% Cell type:code id: tags:
-
-``` python
-c.plugins()
-```
-
-%% Cell type:markdown id: tags:
-
-Or, more concisely:
-
-%% Cell type:code id: tags:
-
-``` python
-c.plugins().keys()
-```
-
-%% Cell type:markdown id: tags:
-
-Local Endpoint
---------------
-
-%% Cell type:markdown id: tags:
-
-To run your own instance of senpy, just create a docker container with the latest Senpy image. Using `--default-plugins` you will get some extra plugins to start playing with the API.
-
-%% Cell type:code id: tags:
-
-``` python
-!docker run -ti --name 'SenpyEndpoint' -d -p 6000:5000 gsiupm/senpy:0.8.6 --host 0.0.0.0 --default-plugins
-```
-
-%% Cell type:markdown id: tags:
-
-To use this endpoint:
-
-%% Cell type:code id: tags:
-
-``` python
-c_local = Client('http://127.0.0.1:6000/api')
-```
-
-%% Cell type:markdown id: tags:
-
-That's all! After you are done with your analysis, stop the docker container:
-
-%% Cell type:code id: tags:
-
-``` python
-!docker stop SenpyEndpoint
-!docker rm SenpyEndpoint
-```

docs/SenpyClientUse.rst

-
-Client
-======
-
-Demo Endpoint
--------------
-
-Import Client and send a request
-
-.. code:: python
-
-    from senpy.client import Client
-    
-    c = Client('http://latest.senpy.cluster.gsi.dit.upm.es/api')
-    r = c.analyse('I like Pizza', algorithm='sentiment140')
-
-Print response
-
-.. code:: python
-
-    for entry in r.entries:
-          print('{} -> {}'.format(entry['text'], entry['sentiments'][0]['marl:hasPolarity']))
-
-
-.. parsed-literal::
-
-    I like Pizza -> marl:Positive
-
-
-Obtain a list of available plugins
-
-.. code:: python
-
-    for plugin in c.request('/plugins')['plugins']:
-        print(plugin['name'])
-
-
-.. parsed-literal::
-
-    emoRand
-    rand
-    sentiment140
-
-
-Local Endpoint
---------------
-
-Run a docker container with Senpy image and default plugins
-
-.. code::
-
-    docker run -ti --name 'SenpyEndpoint' -d -p 5000:5000 gsiupm/senpy:0.8.6 --host 0.0.0.0 --default-plugins
-
-
-.. parsed-literal::
-
-    a0157cd98057072388bfebeed78a830da7cf0a796f4f1a3fd9188f9f2e5fe562
-
-
-Import client and send a request to localhost
-
-.. code:: python
-
-    c_local = Client('http://127.0.0.1:5000/api')
-    r = c_local.analyse('Hello world', algorithm='sentiment140')
-
-Print response
-
-.. code:: python
-
-    for entry in r.entries:
-          print('{} -> {}'.format(entry['text'], entry['sentiments'][0]['marl:hasPolarity']))
-
-
-.. parsed-literal::
-
-    Hello world -> marl:Neutral
-
-
-Obtain a list of available plugins deployed locally
-
-.. code:: python
-
-    c_local.plugins().keys()
-
-
-.. parsed-literal::
-
-    rand
-    sentiment140
-    emoRand
-
-
-Stop the docker container
-
-.. code:: python
-
-    !docker stop SenpyEndpoint
-    !docker rm SenpyEndpoint
-
-
-.. parsed-literal::
-
-    SenpyEndpoint
-    SenpyEndpoint
-

docs/advanced.rst

+Advanced usage
+--------------
+
+.. toctree::
+    :maxdepth: 1
+
+    server-cli
+    conversion
+    commandline
\ No newline at end of file

docs/api.rst

@@ -25,7 +25,7 @@ NIF API
         "@context":"http://127.0.0.1/api/contexts/Results.jsonld",
         "@id":"_:Results_11241245.22",
         "@type":"results"
-        "analysis": [
+        "activities": [
           "plugins/sentiment-140_0.1" 
         ],
         "entries": [
@@ -73,7 +73,7 @@ NIF API
 .. http:get:: /api/plugins
 
    Returns a list of installed plugins. 
-   **Example request**:
+   **Example request and response**:
 
    .. sourcecode:: http
 
@@ -82,10 +82,6 @@ NIF API
       Accept: application/json, text/javascript
 
 
-   **Example response**:
-
-   .. sourcecode:: http
-
       {
         "@id": "plugins/sentiment-140_0.1", 
         "@type": "sentimentPlugin", 
@@ -143,19 +139,14 @@ NIF API
 .. http:get:: /api/plugins/<pluginname>
 
    Returns the information of a specific plugin.
-   **Example request**:
+   **Example request and response**:
 
    .. sourcecode:: http
 
-      GET /api/plugins/rand/ HTTP/1.1
+      GET /api/plugins/sentiment-random/ HTTP/1.1
       Host: localhost
       Accept: application/json, text/javascript
 
-
-   **Example response**:
-
-   .. sourcecode:: http
-
       {
         "@context": "http://127.0.0.1/api/contexts/ExamplePlugin.jsonld", 
         "@id": "plugins/ExamplePlugin_0.1", 

docs/apischema.rst

-API and Examples
-################
+API and vocabularies
+####################
+
 .. toctree::
 
    vocabularies.rst

docs/bad-examples/results/example-analysis-as-id-FAIL.json

@@ -2,7 +2,7 @@
   "@context": "http://mixedemotions-project.eu/ns/context.jsonld",
   "@id": "me:Result1",
   "@type": "results",
-  "analysis": [
+  "activities": [
     "me:SAnalysis1",
     "me:SgAnalysis1",
     "me:EmotionAnalysis1",

docs/bad-examples/results/example-basic-FAIL.json

@@ -2,7 +2,7 @@
   "@context": "http://mixedemotions-project.eu/ns/context.jsonld",
   "@id": "http://example.com#NIFExample",
   "@type": "results",
-  "analysis": [
+  "activities": [
   ],
   "entries": [
     {

docs/commandline.rst

 Command line
 ============
 
-This video shows how to analyse text directly on the command line using the senpy tool.
+Although the main use of senpy is to publish services, the tool can also be used locally to analyze text in the command line.
+This is a short video demonstration:
 
 .. image:: https://asciinema.org/a/9uwef1ghkjk062cw2t4mhzpyk.png
    :width: 100%

docs/conversion.rst

@@ -7,9 +7,9 @@ Senpy includes experimental support for emotion/sentiment conversion plugins.
 Use
 ===
 
-Consider the original query: http://127.0.0.1:5000/api/?i=hello&algo=emoRand
+Consider the original query: http://127.0.0.1:5000/api/?i=hello&algo=emotion-random
 
-The requested plugin (emoRand) returns emotions using Ekman's model (or big6 in EmotionML):
+The requested plugin (emotion-random) returns emotions using Ekman's model (or big6 in EmotionML):
 
 .. code:: json
 
@@ -21,14 +21,14 @@ The requested plugin (emoRand) returns emotions using Ekman's model (or big6 in
                 "@type": "emotion",
                 "onyx:hasEmotionCategory": "emoml:big6anger"
             },
-            "prov:wasGeneratedBy": "plugins/emoRand_0.1"
+            "prov:wasGeneratedBy": "plugins/emotion-random_0.1"
           }
 
           
 
 To get these emotions in VAD space (FSRE dimensions in EmotionML), we'd do this:
 
-http://127.0.0.1:5000/api/?i=hello&algo=emoRand&emotionModel=emoml:fsre-dimensions
+http://127.0.0.1:5000/api/?i=hello&algo=emotion-random&emotionModel=emoml:fsre-dimensions
 
 This call, provided there is a valid conversion plugin from Ekman's to VAD, would return something like this:
 
@@ -42,7 +42,7 @@ This call, provided there is a valid conversion plugin from Ekman's to VAD, woul
                 "@type": "emotion",
                 "onyx:hasEmotionCategory": "emoml:big6anger"
                 },
-            "prov:wasGeneratedBy": "plugins/emoRand_0.1"
+            "prov:wasGeneratedBy": "plugins/emotion-random.1"
           }, {
             "@type": "emotionSet",
             "onyx:hasEmotion": {
@@ -69,7 +69,7 @@ It is also possible to get the original emotion nested within the new converted
                 "@type": "emotion",
                 "onyx:hasEmotionCategory": "emoml:big6anger"
                 },
-            "prov:wasGeneratedBy": "plugins/emoRand_0.1"
+            "prov:wasGeneratedBy": "plugins/emotion-random.1"
             "onyx:wasDerivedFrom": {
                 "@type": "emotionSet",
                 "onyx:hasEmotion": {

docs/demo.rst

 Demo
 ----
 
-There is a demo available on http://senpy.cluster.gsi.dit.upm.es/, where you can test a serie of different plugins.
+There is a demo available on http://senpy.gsi.upm.es/, where you can test a live instance of Senpy, with several open source plugins.
 You can use the playground (a web interface) or make HTTP requests to the service API.
 
 .. image:: senpy-playground.png