Commit 8a516d92 authored by J. Fernando Sánchez's avatar J. Fernando Sánchez
Browse files

Multiple changes in the API, schemas and UI

Check out the CHANGELOG.md file for more information
parent 4ba30304
Developing new services
-----------------------
Developing web services can be hard.
To illustrate it, the figure below summarizes the typical features in a text analysis service.
.. image:: senpy-framework.png
:width: 60%
:align: center
Senpy implements all the common blocks, so developers can focus on what really matters: great analysis algorithms that solve real problems.
Among other things, Senpy takes care of these tasks:
* Interfacing with the user: parameter validation, error handling.
* Formatting: JSON-LD, Turtle/n-triples input and output, or simple text input
* Linked Data: senpy results are semantically annotated, using a series of well established vocabularies, and sane default URIs.
* User interface: a web UI where users can explore your service and test different settings
* A client to interact with the service. Currently only available in Python.
You only need to provide the algorithm to turn a piece of text into an annotation
Sharing your sentiment analysis with the world has never been easier!
.. toctree::
:maxdepth: 1
plugins-quickstart
plugins-faq
\ No newline at end of file
Examples
------
--------
All the examples in this page use the :download:`the main schema <_static/schemas/definitions.json>`.
Simple NIF annotation
......@@ -17,6 +18,7 @@ Sentiment Analysis
.....................
Description
,,,,,,,,,,,
This annotation corresponds to the sentiment analysis of an input. The example shows the sentiment represented according to Marl format.
The sentiments detected are contained in the Sentiments array with their related part of the text.
......
......@@ -2,7 +2,7 @@
"@context": "http://mixedemotions-project.eu/ns/context.jsonld",
"@id": "me:Result1",
"@type": "results",
"analysis": [
"activities": [
{
"@id": "_:SAnalysis1_Activity",
"@type": "marl:SentimentAnalysis",
......
......@@ -2,7 +2,7 @@
"@context": "http://mixedemotions-project.eu/ns/context.jsonld",
"@id": "me:Result1",
"@type": "results",
"analysis": [ ],
"activities": [ ],
"entries": [
{
"@id": "http://example.org#char=0,40",
......
......@@ -2,7 +2,7 @@
"@context": "http://mixedemotions-project.eu/ns/context.jsonld",
"@id": "me:Result1",
"@type": "results",
"analysis": [
"activities": [
{
"@id": "_:SAnalysis1_Activity",
"@type": "marl:SentimentAnalysis",
......
......@@ -2,7 +2,7 @@
"@context": "http://mixedemotions-project.eu/ns/context.jsonld",
"@id": "me:Result1",
"@type": "results",
"analysis": [
"activities": [
{
"@id": "me:EmotionAnalysis1_Activity",
"@type": "me:EmotionAnalysis1",
......
......@@ -2,7 +2,7 @@
"@context": "http://mixedemotions-project.eu/ns/context.jsonld",
"@id": "me:Result1",
"@type": "results",
"analysis": [
"activities": [
{
"@id": "_:NER1_Activity",
"@type": "me:NERAnalysis",
......
......@@ -7,7 +7,7 @@
],
"@id": "me:Result1",
"@type": "results",
"analysis": [
"activities": [
{
"@id": "me:HesamsAnalysis_Activity",
"@type": "onyx:EmotionAnalysis",
......
......@@ -2,7 +2,7 @@
"@context": "http://mixedemotions-project.eu/ns/context.jsonld",
"@id": "me:Result1",
"@type": "results",
"analysis": [
"activities": [
{
"@id": "_:SAnalysis1_Activity",
"@type": "marl:SentimentAnalysis",
......
......@@ -2,7 +2,7 @@
"@context": "http://mixedemotions-project.eu/ns/context.jsonld",
"@id": "me:Result1",
"@type": "results",
"analysis": [
"activities": [
{
"@id": "_:SgAnalysis1_Activity",
"@type": "me:SuggestionAnalysis",
......
......@@ -4,32 +4,31 @@ Welcome to Senpy's documentation!
:target: http://senpy.readthedocs.io/en/latest/
.. image:: https://badge.fury.io/py/senpy.svg
:target: https://badge.fury.io/py/senpy
.. image:: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/badges/master/build.svg
:target: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/commits/master
.. image:: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/badges/master/coverage.svg
:target: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/commits/master
.. image:: https://lab.gsi.upm.es/senpy/senpy/badges/master/build.svg
:target: https://lab.gsi.upm.es/senpy/senpy/commits/master
.. image:: https://lab.gsi.upm.es/senpy/senpy/badges/master/coverage.svg
:target: https://lab.gsi.upm.es/senpy/senpy/commits/master
.. image:: https://img.shields.io/pypi/l/requests.svg
:target: https://lab.cluster.gsi.dit.upm.es/senpy/senpy/
:target: https://lab.gsi.upm.es/senpy/senpy/
Senpy is a framework for sentiment and emotion analysis services.
Services built with senpy are interchangeable and easy to use because they share a common :doc:`apischema`.
It also simplifies service development.
Senpy services are interchangeable and easy to use because they share a common semantic :doc:`apischema`.
.. image:: senpy-architecture.png
:width: 100%
:align: center
If you interested in consuming Senpy services, read :doc:`Quickstart`.
To get familiar with the concepts behind Senpy, and what it can offer for service developers, check out :doc:`development`.
:doc:`apischema` contains information about the semantic models and vocabularies used by Senpy.
.. toctree::
:caption: Learn more about senpy:
:maxdepth: 2
senpy
Quickstart
installation
demo
usage
development
apischema
plugins
conversion
about
advanced
demo
publications
......@@ -32,27 +32,25 @@ If you want to install senpy globally, use sudo instead of the ``--user`` flag.
Docker Image
************
Build the image or use the pre-built one:
The base image of senpy comes with some builtin plugins that you can use:
.. code:: bash
docker run -ti -p 5000:5000 gsiupm/senpy --host 0.0.0.0 --default-plugins
docker run -ti -p 5000:5000 gsiupm/senpy --host 0.0.0.0
To add custom plugins, use a docker volume:
To add your custom plugins, you can use a docker volume:
.. code:: bash
docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins gsiupm/senpy --host 0.0.0.0 --default-plugins -f /plugins
docker run -ti -p 5000:5000 -v <PATH OF PLUGINS>:/plugins gsiupm/senpy --host 0.0.0.0 --plugins -f /plugins
Python 2
........
There is a Senpy version for python2 too:
There is a Senpy image for **python 2**, too:
.. code:: bash
docker run -ti -p 5000:5000 gsiupm/senpy:python2.7 --host 0.0.0.0 --default-plugins
docker run -ti -p 5000:5000 gsiupm/senpy:python2.7 --host 0.0.0.0
Alias
......@@ -62,7 +60,7 @@ If you are using the docker approach regularly, it is advisable to use a script
.. code:: bash
alias senpy='docker run --rm -ti -p 5000:5000 -v $PWD:/senpy-plugins gsiupm/senpy --default-plugins'
alias senpy='docker run --rm -ti -p 5000:5000 -v $PWD:/senpy-plugins gsiupm/senpy'
Now, you may run senpy from any folder in your computer like so:
......
......@@ -110,4 +110,4 @@ Now, in a file named ``helloworld.py``:
entry.sentiments.append(sentiment)
yield entry
The complete code of the example plugin is available `here <https://lab.cluster.gsi.dit.upm.es/senpy/plugin-prueba>`__.
The complete code of the example plugin is available `here <https://lab.gsi.upm.es/senpy/plugin-prueba>`__.
Developing new plugins
----------------------
This document contains the minimum to get you started with developing new analysis plugin.
For an example of conversion plugins, see :doc:`conversion`.
For a description of definition files, see :doc:`plugins-definition`.
A more step-by-step tutorial with slides is available `here <https://lab.cluster.gsi.dit.upm.es/senpy/senpy-tutorial>`__
F.A.Q.
======
.. contents:: :local:
What is a plugin?
=================
A plugin is a python object that can process entries. Given an entry, it will modify it, add annotations to it, or generate new entries.
What is an entry?
=================
Entries are objects that can be annotated.
In general, they will be a piece of text.
By default, entries are `NIF contexts <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core/nif-core.html>`_ represented in JSON-LD format.
It is a dictionary/JSON object that looks like this:
.. code:: python
{
"@id": "<unique identifier or blank node name>",
"nif:isString": "input text",
"sentiments": [ {
...
}
],
...
}
Annotations are added to the object like this:
.. code:: python
entry = Entry()
entry.vocabulary__annotationName = 'myvalue'
entry['vocabulary:annotationName'] = 'myvalue'
entry['annotationNameURI'] = 'myvalue'
Where vocabulary is one of the prefixes defined in the default senpy context, and annotationURI is a full URI.
The value may be any valid JSON-LD dictionary.
For simplicity, senpy includes a series of models by default in the ``senpy.models`` module.
What are annotations?
=====================
#####################
They are objects just like entries.
Senpy ships with several default annotations, including: ``Sentiment``, ``Emotion``, ``EmotionSet``...jk bb
What's a plugin made of?
========================
########################
When receiving a query, senpy selects what plugin or plugins should process each entry, and in what order.
It also makes sure the every entry and the parameters provided by the user meet the plugin requirements.
......@@ -73,37 +30,25 @@ In practice, this is what a plugin looks like, tests included:
The lines highlighted contain some information about the plugin.
In particular, the following information is mandatory:
* A unique name for the class. In our example, Rand.
* A unique name for the class. In our example, sentiment-random.
* The subclass/type of plugin. This is typically either `SentimentPlugin` or `EmotionPlugin`. However, new types of plugin can be created for different annotations. The only requirement is that these new types inherit from `senpy.Analysis`
* A description of the plugin. This can be done simply by adding a doc to the class.
* A version, which should get updated.
* An author name.
Plugins Code
============
The basic methods in a plugin are:
* analyse_entry: called in every user requests. It takes two parameters: ``Entry``, the entry object, and ``params``, the parameters supplied by the user. It should yield one or more ``Entry`` objects.
* activate: used to load memory-hungry resources. For instance, to train a classifier.
* deactivate: used to free up resources when the plugin is no longer needed.
Plugins are loaded asynchronously, so don't worry if the activate method takes too long. The plugin will be marked as activated once it is finished executing the method.
How does senpy find modules?
============================
############################
Senpy looks for files of two types:
* Python files of the form `senpy_<NAME>.py` or `<NAME>_plugin.py`. In these files, it will look for: 1) Instances that inherit from `senpy.Plugin`, or subclasses of `senpy.Plugin` that can be initialized without a configuration file. i.e. classes that contain all the required attributes for a plugin.
* Plugin definition files (see :doc:`advanced-plugins`)
* Plugin definition files (see :doc:`plugins-definition`)
Defining additional parameters
==============================
How can I define additional parameters for my plugin?
#####################################################
Your plugin may ask for additional parameters from the users of the service by using the attribute ``extra_params`` in your plugin definition.
Your plugin may ask for additional parameters from users by using the attribute ``extra_params`` in your plugin definition.
It takes a dictionary, where the keys are the name of the argument/parameter, and the value has the following fields:
* aliases: the different names which can be used in the request to use the parameter.
......@@ -124,8 +69,8 @@ It takes a dictionary, where the keys are the name of the argument/parameter, an
Loading data and files
======================
How should I load external data and files
#########################################
Most plugins will need access to files (dictionaries, lexicons, etc.).
These files are usually heavy or under a license that does not allow redistribution.
......@@ -144,7 +89,7 @@ Plugins have a convenience function `self.open` which will automatically prepend
file_in_data = <FILE PATH>
file_in_sources = <FILE PATH>
def activate(self):
def on activate(self):
with self.open(self.file_in_data) as f:
self._classifier = train_from_file(f)
file_in_source = os.path.join(self.get_folder(), self.file_in_sources)
......@@ -155,8 +100,8 @@ Plugins have a convenience function `self.open` which will automatically prepend
It is good practice to specify the paths of these files in the plugin configuration, so the same code can be reused with different resources.
Docker image
============
Can I build a docker image for my plugin?
#########################################
Add the following dockerfile to your project to generate a docker image with your plugin:
......@@ -204,17 +149,15 @@ Adding data to the image:
FROM gsiupm/senpy:1.0.1
COPY data /
F.A.Q.
======
What annotations can I use?
???????????????????????????
###########################
You can add almost any annotation to an entry.
The most common use cases are covered in the :doc:`apischema`.
Why does the analyse function yield instead of return?
??????????????????????????????????????????????????????
######################################################
This is so that plugins may add new entries to the response or filter some of them.
For instance, a chunker may split one entry into several.
......@@ -222,7 +165,7 @@ On the other hand, a conversion plugin may leave out those entries that do not c
If I'm using a classifier, where should I train it?
???????????????????????????????????????????????????
###################################################
Training a classifier can be time time consuming. To avoid running the training unnecessarily, you can use ShelfMixin to store the classifier. For instance:
......@@ -256,7 +199,7 @@ A corrupt shelf prevents the plugin from loading.
If you do not care about the data in the shelf, you can force your plugin to remove the corrupted file and load anyway, set the 'force_shelf' to True in your plugin and start it again.
How can I turn an external service into a plugin?
?????????????????????????????????????????????????
#################################################
This example ilustrate how to implement a plugin that accesses the Sentiment140 service.
......@@ -292,8 +235,8 @@ This example ilustrate how to implement a plugin that accesses the Sentiment140
yield entry
Can I activate a DEBUG mode for my plugin?
???????????????????????????????????????????
How can I activate a DEBUG mode for my plugin?
###############################################
You can activate the DEBUG mode by the command-line tool using the option -d.
......@@ -309,6 +252,6 @@ Additionally, with the ``--pdb`` option you will be dropped into a pdb post mort
python -m pdb yourplugin.py
Where can I find more code examples?
????????????????????????????????????
####################################
See: `<http://github.com/gsi-upm/senpy-plugins-community>`_.
Quickstart for service developers
=================================
This document contains the minimum to get you started with developing new services using Senpy.
For an example of conversion plugins, see :doc:`conversion`.
For a description of definition files, see :doc:`plugins-definition`.
A more step-by-step tutorial with slides is available `here <https://lab.gsi.upm.es/senpy/senpy-tutorial>`__
.. contents:: :local:
Installation
############
First of all, you need to install the package.
See :doc:`installation` for instructions.
Once installed, the `senpy` command should be available.
Architecture
############
The main component of a sentiment analysis service is the algorithm itself. However, for the algorithm to work, it needs to get the appropriate parameters from the user, format the results according to the defined API, interact with the user whn errors occur or more information is needed, etc.
Senpy proposes a modular and dynamic architecture that allows:
* Implementing different algorithms in a extensible way, yet offering a common interface.
* Offering common services that facilitate development, so developers can focus on implementing new and better algorithms.
The framework consists of two main modules: Senpy core, which is the building block of the service, and Senpy plugins, which consist of the analysis algorithm. The next figure depicts a simplified version of the processes involved in an analysis with the Senpy framework.
.. image:: senpy-architecture.png
:width: 100%
:align: center
What is a plugin?
#################
A plugin is a python object that can process entries. Given an entry, it will modify it, add annotations to it, or generate new entries.
What is an entry?
#################
Entries are objects that can be annotated.
In general, they will be a piece of text.
By default, entries are `NIF contexts <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core/nif-core.html>`_ represented in JSON-LD format.
It is a dictionary/JSON object that looks like this:
.. code:: python
{
"@id": "<unique identifier or blank node name>",
"nif:isString": "input text",
"sentiments": [ {
...
}
],
...
}
Annotations are added to the object like this:
.. code:: python
entry = Entry()
entry.vocabulary__annotationName = 'myvalue'
entry['vocabulary:annotationName'] = 'myvalue'
entry['annotationNameURI'] = 'myvalue'
Where vocabulary is one of the prefixes defined in the default senpy context, and annotationURI is a full URI.
The value may be any valid JSON-LD dictionary.
For simplicity, senpy includes a series of models by default in the ``senpy.models`` module.
Plugins Code
############
The basic methods in a plugin are:
* analyse_entry: called in every user requests. It takes two parameters: ``Entry``, the entry object, and ``params``, the parameters supplied by the user. It should yield one or more ``Entry`` objects.
* activate: used to load memory-hungry resources. For instance, to train a classifier.
* deactivate: used to free up resources when the plugin is no longer needed.
Plugins are loaded asynchronously, so don't worry if the activate method takes too long. The plugin will be marked as activated once it is finished executing the method.
About
--------
Publications
============
If you use Senpy in your research, please cite `Senpy: A Pragmatic Linked Sentiment Analysis Framework <http://gsi.dit.upm.es/index.php/es/investigacion/publicaciones?view=publication&task=show&id=417>`__ (`BibTex <http://gsi.dit.upm.es/index.php/es/investigacion/publicaciones?controller=publications&task=export&format=bibtex&id=417>`__):
......@@ -9,3 +10,37 @@ If you use Senpy in your research, please cite `Senpy: A Pragmatic Linked Sentim
Senpy: A Pragmatic Linked Sentiment Analysis Framework.
In Data Science and Advanced Analytics (DSAA),
2016 IEEE International Conference on (pp. 735-742). IEEE.
Senpy uses Onyx for emotion representation, first introduced in:
.. code-block:: text
Sánchez-Rada, J. F., & Iglesias, C. A. (2016).
Onyx: A linked data approach to emotion representation.
Information Processing & Management, 52(1), 99-114.
Senpy uses Marl for sentiment representation, which was presented in:
.. code-block:: text
Westerski, A., Iglesias Fernandez, C. A., & Tapia Rico, F. (2011).
Linked opinions: Describing sentiments on the structured web of data.
Senpy has been used extensively in the toolbox of the MixedEmotions project:
.. code-block:: text
Buitelaar, P., Wood, I. D., Arcan, M., McCrae, J. P., Abele, A., Robin, C., … Tummarello, G. (2018).
MixedEmotions: An Open-Source Toolbox for Multi-Modal Emotion Analysis.
IEEE Transactions on Multimedia.
The representation models, formats and challenges are partially covered in a chapter of the book Sentiment Analysis in Social Networks:
.. code-block:: text
Iglesias, C. A., Sánchez-Rada, J. F., Vulcu, G., & Buitelaar, P. (2017).
Linked Data Models for Sentiment and Emotion Analysis in Social Networks.
In Sentiment Analysis in Social Networks (pp. 49-69).
\ No newline at end of file
What is Senpy?
--------------
Senpy is a framework for text analysis using Linked Data. There are three main applications of Senpy so far: sentiment and emotion analysis, user profiling and entity recoginition. Annotations and Services are compliant with NIF (NLP Interchange Format).
Senpy is a framework for sentiment and emotion analysis services.
Its goal is to produce analysis services that are interchangeable and fully interoperable.
Senpy aims at providing a framework where analysis modules can be integrated easily as plugins, and providing a core functionality for managing tasks such as data validation, user interaction, formatting, logging, translation to linked data, etc.
The figure below summarizes the typical features in a text analysis service.
Senpy implements all the common blocks, so developers can focus on what really matters: great analysis algorithms that solve real problems.
.. image:: senpy-framework.png
:width: 60%
.. image:: senpy-architecture.png
:width: 100%
:align: center
Senpy for end users
===================
All services built using senpy share a common interface.
This allows users to use them (almost) interchangeably.
Senpy comes with a :ref:`built-in client`.
Senpy for service developers
============================
This allows users to use them (almost) interchangeably, with the same API and tools, simply by pointing to a different URL or changing a parameter.
The common schema also makes it easier to evaluate the performance of different algorithms and services.
In fact, Senpy has a built-in evaluation API you can use to compare results with different algorithms.
Senpy is a framework that turns your sentiment or emotion analysis algorithm into a full blown semantic service.
Senpy takes care of:
Services can also use the common interface to communicate with each other.
And higher level features can be built on top of these services, such as automatic fusion of results, emotion model conversion, and service discovery.
* Interfacing with the user: parameter validation, error handling.
* Formatting: JSON-LD, Turtle/n-triples input and output, or simple text input
* Linked Data: senpy results are semantically annotated, using a series of well established vocabularies, and sane default URIs.
* User interface: a web UI where users can explore your service and test different settings
* A client to interact with the service. Currently only available in Python.
Sharing your sentiment analysis with the world has never been easier!
Check out the :doc:`plugins` if you have developed an analysis algorithm (e.g. sentiment analysis) and you want to publish it as a service.
Architecture
============
The main component of a sentiment analysis service is the algorithm itself. However, for the algorithm to work, it needs to get the appropriate parameters from the user, format the results according to the defined API, interact with the user whn errors occur or more information is needed, etc.
Senpy proposes a modular and dynamic architecture that allows:
These benefits are not limited to new services.
The community has developed wrappers for some proprietary and commercial services (such as sentiment140 and Meaning Cloud), so you can consult them as.
Senpy comes with a :ref:`built-in client`.
* Implementing different algorithms in a extensible way, yet offering a common interface.
* Offering common services that facilitate development, so developers can focus on implementing new and better algorithms.
The framework consists of two main modules: Senpy core, which is the building block of the service, and Senpy plugins, which consist of the analysis algorithm. The next figure depicts a simplified version of the processes involved in an analysis with the Senpy framework.
To achieve this goal, Senpy uses a Linked Data principled approach, based on the NIF (NLP Interchange Format) specification, and open vocabularies such as Marl and Onyx.
You can learn more about this in :doc:`vocabularies`.
.. image:: senpy-architecture.png
:width: 100%
:align: center
Check out :doc:`plugins` if you have developed an analysis algorithm (e.g. sentiment analysis) and you want to publish it as a service.
......@@ -5,10 +5,11 @@ The senpy server is launched via the `senpy` command:
.. code:: text
usage: senpy [-h] [--level logging_level] [--debug] [--default-plugins]
[--host HOST] [--port PORT] [--plugins-folder PLUGINS_FOLDER]
[--only-install] [--only-list] [--data-folder DATA_FOLDER]
[--threaded] [--version]
usage: senpy [-h] [--level logging_level] [--log-format log_format] [--debug]
[--no-default-plugins] [--host HOST] [--port PORT]
[--plugins-folder PLUGINS_FOLDER] [--only-install] [--only-test]
[--test] [--only-list] [--data-folder DATA_FOLDER]
[--no-threaded] [--no-deps] [--version] [--allow-fail]
Run a Senpy server
......@@ -16,20 +17,25 @@ The senpy server is launched via the `senpy` command:
-h, --help show this help message and exit
--level logging_level, -l logging_level
Logging level
--log-format log_format
Logging format
--debug, -d Run the application in debug mode
--default-plugins Load the default plugins
--no-default-plugins Do not load the default plugins