Commit 21a5a3f2 authored by J. Fernando Sánchez's avatar J. Fernando Sánchez

Macro commit

* Fixed Options for extra_params in UI
* Enhanced meta-programming for models
* Plugins can be imported from a python file if they're named
`senpy_<whatever>.py>` (no need for `.senpy` anymore!)
* Add docstings and tests to most plugins
* Read plugin description from the docstring
* Refactor code to get rid of unnecessary `.senpy`s
* Load models, plugins and utils into the main namespace (see __init__.py)
* Enhanced plugin development/experience with utils (easy_test, easy_serve)
* Fix bug in check_template that wouldn't check objects
* Make model defaults a private variable
* Add option to list loaded plugins in CLI
* Update docs
parent abd401f8
Advanced plugin definition
--------------------------
In addition to finding plugins defined in source code files, senpy can also load a special type of definition file (`.senpy` files).
This used to be the only mechanism for loading in earlier versions of senpy.
The definition file contains basic information
Lastly, it is also possible to add new plugins programmatically.
.. contents:: :local:
What is a plugin?
=================
A plugin is a program that, given a text, will add annotations to it.
In practice, a plugin consists of at least two files:
- Definition file: a `.senpy` file that describes the plugin (e.g. what input parameters it accepts, what emotion model it uses).
- Python module: the actual code that will add annotations to each input.
This separation allows us to deploy plugins that use the same code but employ different parameters.
For instance, one could use the same classifier and processing in several plugins, but train with different datasets.
This scenario is particularly useful for evaluation purposes.
The only limitation is that the name of each plugin needs to be unique.
Definition files
================
The definition file complements and overrides the attributes provided by the plugin.
It can be written in YAML or JSON.
The most important attributes are:
* **name**: unique name that senpy will use internally to identify the plugin.
* **module**: indicates the module that contains the plugin code, which will be automatically loaded by senpy.
* **version**
* extra_params: to add parameters to the senpy API when this plugin is requested. Those parameters may be required, and have aliased names. For instance:
.. code:: yaml
extra_params:
hello_param:
aliases: # required
- hello_param
- hello
required: true
default: Hi you
values:
- Hi you
- Hello y'all
- Howdy
A complete example:
.. code:: yaml
name: <Name of the plugin>
module: <Python file>
version: 0.1
And the json equivalent:
.. code:: json
{
"name": "<Name of the plugin>",
"module": "<Python file>",
"version": "0.1"
}
Example plugin with a definition file
=====================================
In this section, we will implement a basic sentiment analysis plugin.
To determine the polarity of each entry, the plugin will compare the length of the string to a threshold.
This threshold will be included in the definition file.
The definition file would look like this:
.. code:: yaml
name: helloworld
module: helloworld
version: 0.0
threshold: 10
description: Hello World
Now, in a file named ``helloworld.py``:
.. code:: python
#!/bin/env python
#helloworld.py
from senpy import AnalysisPlugin
from senpy import Sentiment
class HelloWorld(AnalysisPlugin):
def analyse_entry(entry, params):
'''Basically do nothing with each entry'''
sentiment = Sentiment()
if len(entry.text) < self.threshold:
sentiment['marl:hasPolarity'] = 'marl:Positive'
else:
sentiment['marl:hasPolarity'] = 'marl:Negative'
entry.sentiments.append(sentiment)
yield entry
The complete code of the example plugin is available `here <https://lab.cluster.gsi.dit.upm.es/senpy/plugin-prueba>`__.
This diff is collapsed.
......@@ -7,21 +7,29 @@ The senpy server is launched via the `senpy` command:
usage: senpy [-h] [--level logging_level] [--debug] [--default-plugins]
[--host HOST] [--port PORT] [--plugins-folder PLUGINS_FOLDER]
[--only-install]
[--only-install] [--only-list] [--data-folder DATA_FOLDER]
[--threaded] [--version]
Run a Senpy server
optional arguments:
-h, --help show this help message and exit
--level logging_level, -l logging_level
-h, --help show this help message and exit
--level logging_level, -l logging_level
Logging level
--debug, -d Run the application in debug mode
--default-plugins Load the default plugins
--host HOST Use 0.0.0.0 to accept requests from any host.
--port PORT, -p PORT Port to listen on.
--plugins-folder PLUGINS_FOLDER, -f PLUGINS_FOLDER
--debug, -d Run the application in debug mode
--default-plugins Load the default plugins
--host HOST Use 0.0.0.0 to accept requests from any host.
--port PORT, -p PORT Port to listen on.
--plugins-folder PLUGINS_FOLDER, -f PLUGINS_FOLDER
Where to look for plugins.
--only-install, -i Do not run a server, only install plugin dependencies
--only-install, -i Do not run a server, only install plugin dependencies
--only-list, --list Do not run a server, only list plugins found
--data-folder DATA_FOLDER, --data DATA_FOLDER
Where to look for data. It be set with the SENPY_DATA
environment variable as well.
--threaded Run a threaded server
--version, -v Output the senpy version and exit
When launched, the server will recursively look for plugins in the specified plugins folder (the current working directory by default).
......
This is a collection of plugins that exemplify certain aspects of plugin development with senpy.
In ascending order of complexity, there are:
* Basic: a very basic analysis that does sentiment analysis based on emojis.
* Configurable: a version of `basic` with a configurable map of emojis for each sentiment.
* Parameterized: like `basic_info`, but users set the map in each query (via `extra_parameters`).
* mynoop: shows how to add a definition file with external requirements for a plugin. Doing this with a python-only module would require moving all imports of the requirements to their functions, which is considered bad practice.
* Async: a barebones example of training a plugin and analyzing data in parallel.
All of the plugins in this folder include a set of test cases and they are periodically tested with the latest version of senpy.
Additioanlly, for an example of stand-alone plugin that can be tested and deployed with docker, take a look at: lab.cluster.gsi.dit.upm.es/senpy/plugin-example
bbm
from senpy.plugins import AnalysisPlugin
from senpy import AnalysisPlugin
import multiprocessing
......@@ -7,10 +7,15 @@ def _train(process_number):
return process_number
class AsyncPlugin(AnalysisPlugin):
class Async(AnalysisPlugin):
'''An example of an asynchronous module'''
author = '@balkian'
version = '0.2'
async = True
def _do_async(self, num_processes):
pool = multiprocessing.Pool(processes=num_processes)
values = pool.map(_train, range(num_processes))
values = sorted(pool.map(_train, range(num_processes)))
return values
......@@ -22,5 +27,11 @@ class AsyncPlugin(AnalysisPlugin):
entry.async_values = values
yield entry
def test(self):
pass
test_cases = [
{
'input': 'any',
'expected': {
'async_values': [0, 1]
}
}
]
---
name: Async
module: asyncplugin
description: I am async
author: "@balkian"
version: '0.1'
async: true
extra_params: {}
\ No newline at end of file
#!/usr/local/bin/python
# coding: utf-8
emoticons = {
'marl:Positive': [':)', ':]', '=)', ':D'],
'marl:Negative': [':(', ':[', '=(']
}
emojis = {
'marl:Positive': ['😁', '😂', '😃', '😄', '😆', '😅', '😄' '😍'],
'marl:Negative': ['😢', '😡', '😠', '😞', '😖', '😔', '😓', '😒']
}
def get_polarity(text, dictionaries=[emoticons, emojis]):
polarity = 'marl:Neutral'
for dictionary in dictionaries:
for label, values in dictionary.items():
for emoticon in values:
if emoticon and emoticon in text:
polarity = label
break
return polarity
#!/usr/local/bin/python
# coding: utf-8
from senpy import easy_test, models, plugins
import basic
class Basic(plugins.SentimentPlugin):
'''Provides sentiment annotation using a lexicon'''
author = '@balkian'
version = '0.1'
def analyse_entry(self, entry, params):
polarity = basic.get_polarity(entry.text)
s = models.Sentiment(marl__hasPolarity=polarity)
s.prov(self)
entry.sentiments.append(s)
yield entry
test_cases = [{
'input': 'Hello :)',
'polarity': 'marl:Positive'
}, {
'input': 'So sad :(',
'polarity': 'marl:Negative'
}, {
'input': 'Yay! Emojis 😁',
'polarity': 'marl:Positive'
}, {
'input': 'But no emoticons 😢',
'polarity': 'marl:Negative'
}]
if __name__ == '__main__':
easy_test()
#!/usr/local/bin/python
# coding: utf-8
from senpy import easy_test, models, plugins
import basic
class Dictionary(plugins.SentimentPlugin):
'''Sentiment annotation using a configurable lexicon'''
author = '@balkian'
version = '0.2'
dictionaries = [basic.emojis, basic.emoticons]
def analyse_entry(self, entry, params):
polarity = basic.get_polarity(entry.text, self.dictionaries)
s = models.Sentiment(marl__hasPolarity=polarity)
s.prov(self)
entry.sentiments.append(s)
yield entry
test_cases = [{
'input': 'Hello :)',
'polarity': 'marl:Positive'
}, {
'input': 'So sad :(',
'polarity': 'marl:Negative'
}, {
'input': 'Yay! Emojis 😁',
'polarity': 'marl:Positive'
}, {
'input': 'But no emoticons 😢',
'polarity': 'marl:Negative'
}]
class EmojiOnly(Dictionary):
'''Sentiment annotation with a basic lexicon of emojis'''
description = 'A plugin'
dictionaries = [basic.emojis]
test_cases = [{
'input': 'Hello :)',
'polarity': 'marl:Neutral'
}, {
'input': 'So sad :(',
'polarity': 'marl:Neutral'
}, {
'input': 'Yay! Emojis 😁',
'polarity': 'marl:Positive'
}, {
'input': 'But no emoticons 😢',
'polarity': 'marl:Negative'
}]
class EmoticonsOnly(Dictionary):
'''Sentiment annotation with a basic lexicon of emoticons'''
dictionaries = [basic.emoticons]
test_cases = [{
'input': 'Hello :)',
'polarity': 'marl:Positive'
}, {
'input': 'So sad :(',
'polarity': 'marl:Negative'
}, {
'input': 'Yay! Emojis 😁',
'polarity': 'marl:Neutral'
}, {
'input': 'But no emoticons 😢',
'polarity': 'marl:Neutral'
}]
class Salutes(Dictionary):
'''Sentiment annotation with a custom lexicon, for illustration purposes'''
dictionaries = [{
'marl:Positive': ['Hello', '!'],
'marl:Negative': ['sad', ]
}]
test_cases = [{
'input': 'Hello :)',
'polarity': 'marl:Positive'
}, {
'input': 'So sad :(',
'polarity': 'marl:Negative'
}, {
'input': 'Yay! Emojis 😁',
'polarity': 'marl:Positive'
}, {
'input': 'But no emoticons 😢',
'polarity': 'marl:Neutral'
}]
if __name__ == '__main__':
easy_test()
from senpy.plugins import SentimentPlugin
from senpy import AnalysisPlugin, easy
class DummyNoInfo(SentimentPlugin):
description = 'This is a dummy self-contained plugin'
class Dummy(AnalysisPlugin):
'''This is a dummy self-contained plugin'''
author = '@balkian'
version = '0.1'
......@@ -13,15 +12,14 @@ class DummyNoInfo(SentimentPlugin):
yield entry
test_cases = [{
"entry": {
"nif:isString": "Hello world!"
'entry': {
'nif:isString': 'Hello',
},
"expected": [{
"nif:isString": "!dlrow olleH"
}]
'expected': {
'nif:isString': 'olleH'
}
}]
if __name__ == '__main__':
d = DummyNoInfo()
d.test()
easy()
from senpy.plugins import SentimentPlugin
class DummyPlugin(SentimentPlugin):
def analyse_entry(self, entry, params):
entry['nif:isString'] = entry['nif:isString'][::-1]
entry.reversed = entry.get('reversed', 0) + 1
yield entry
def test(self):
pass
{
"name": "Dummy",
"module": "dummy",
"description": "I am dummy",
"author": "@balkian",
"version": "0.1",
"extra_params": {
"example": {
"@id": "example_parameter",
"aliases": ["example", "ex"],
"required": false,
"default": 0
}
}
}
name: DummyNoInfo
module: dummy_noinfo
{
"name": "DummyRequired",
"module": "dummy",
"description": "I am dummy",
"author": "@balkian",
"version": "0.1",
"extra_params": {
"example": {
"@id": "example_parameter",
"aliases": ["example", "ex"],
"required": true
}
}
}
from senpy import AnalysisPlugin, easy
class DummyRequired(AnalysisPlugin):
'''This is a dummy self-contained plugin'''
author = '@balkian'
version = '0.1'
extra_params = {
'example': {
'description': 'An example parameter',
'required': True,
'options': ['a', 'b']
}
}
def analyse_entry(self, entry, params):
entry['nif:isString'] = entry['nif:isString'][::-1]
entry.reversed = entry.get('reversed', 0) + 1
yield entry
test_cases = [{
'entry': {
'nif:isString': 'Hello',
},
'expected': None
}, {
'entry': {
'nif:isString': 'Hello',
},
'params': {
'example': 'a'
},
'expected': {
'nif:isString': 'olleH'
}
}]
if __name__ == '__main__':
easy()
import noop
from senpy.plugins import SentimentPlugin
class NoOp(SentimentPlugin):
'''This plugin does nothing. Literally nothing.'''
version = 0
def analyse_entry(self, entry, *args, **kwargs):
yield entry
def test(self):
print(dir(noop))
super(NoOp, self).test()
test_cases = [{
'entry': {
'nif:isString': 'hello'
},
'expected': {
'nif:isString': 'hello'
}
}]
module: mynoop
requirements:
- noop
\ No newline at end of file
from senpy.plugins import SentimentPlugin
class NoOp(SentimentPlugin):
import noop
#!/usr/local/bin/python
# coding: utf-8
from senpy import easy_test, models, plugins
import basic
class ParameterizedDictionary(plugins.SentimentPlugin):
description = 'This is a basic self-contained plugin'
author = '@balkian'
version = '0.2'
extra_params = {
'positive-words': {
'description': 'Comma-separated list of words that are considered positive',
'aliases': ['positive'],
'required': True
},
'negative-words': {
'description': 'Comma-separated list of words that are considered negative',
'aliases': ['negative'],
'required': False
}
}
def analyse_entry(self, entry, params):
positive_words = params['positive-words'].split(',')
negative_words = params['negative-words'].split(',')
dictionary = {
'marl:Positive': positive_words,
'marl:Negative': negative_words,
}
polarity = basic.get_polarity(entry.text, [dictionary])
s = models.Sentiment(marl__hasPolarity=polarity)
s.prov(self)
entry.sentiments.append(s)
yield entry
test_cases = [
{
'input': 'Hello :)',
'polarity': 'marl:Positive',
'parameters': {
'positive': "Hello,:)",
'negative': "sad,:()"
}
},
{
'input': 'Hello :)',
'polarity': 'marl:Negative',
'parameters': {
'positive': "",
'negative': "Hello"
}
}
]
if __name__ == '__main__':
easy_test()
......@@ -2,7 +2,20 @@ from senpy.plugins import AnalysisPlugin
from time import sleep
class SleepPlugin(AnalysisPlugin):
class Sleep(AnalysisPlugin):
'''Dummy plugin to test async'''
author = "@balkian"
version = "0.2"
timeout = 0.05
extra_params = {
"timeout": {
"@id": "timeout_sleep",
"aliases": ["timeout", "to"],
"required": False,
"default": 0
}
}
def activate(self, *args, **kwargs):
sleep(self.timeout)
......
{
"name": "Sleep",
"module": "sleep",
"description": "I am dummy",
"author": "@balkian",
"version": "0.1",
"timeout": 0.05,
"extra_params": {
"timeout": {
"@id": "timeout_sleep",
"aliases": ["timeout", "to"],
"required": false,
"default": 0
}
}
}
......@@ -28,4 +28,10 @@ logger = logging.getLogger(__name__)
logger.info('Using senpy version: {}'.format(__version__))
from .utils import easy, easy_load, easy_test # noqa: F401
from .models import * # noqa: F401,F403
from .plugins import * # noqa: F401,F403
from .extensions import * # noqa: F401,F403
__all__ = ['api', 'blueprints', 'cli', 'extensions', 'models', 'plugins']
......@@ -39,7 +39,7 @@ def main():
'-l',
metavar='logging_level',
type=str,
default="INFO",
default="ERROR",
help='Logging level')
parser.add_argument(
'--debug',
......@@ -75,6 +75,12 @@ def main():
action='store_true',
default=False,
help='Do not run a server, only install plugin dependencies')
parser.add_argument(
'--only-list',
'--list',
action='store_true',
default=False,
help='Do not run a server, only list plugins found')
parser.add_argument(
'--data-folder',
'--data',
......@@ -97,7 +103,6 @@ def main():
print('Senpy version {}'.format(senpy.__version__))
print(sys.version)
exit(1)
logging.basicConfig()
rl = logging.getLogger()
rl.setLevel(getattr(logging, args.level))
app = Flask(__name__)
......@@ -105,6 +110,14 @@ def main():
sp = Senpy(app, args.plugins_folder,
default_plugins=args.default_plugins,
data_folder=args.data_folder)
if args.only_list:
plugins = sp.plugins()
maxwidth = max(len(x.id) for x in plugins)
for plugin in plugins:
import inspect
fpath = inspect.getfile(plugin.__class__)
print('{: <{width}} @ {}'.format(plugin.id, fpath, width=maxwidth))
return
sp.install_deps()
if args.only_install:
return
......
......@@ -99,7 +99,7 @@ NIF_PARAMS = {
"aliases": ["f"],
"required": False,
"default": "text",
"options": ["turtle", "text", "json-ld"],
"options": ["text", "json-ld"],
},
"language": {
"@id": "language",
......@@ -130,13 +130,11 @@ def parse_params(indict, *specs):
wrong_params = {}
for spec in specs:
for param, options in iteritems(spec):
if param[0] == "@": # Exclude json-ld properties
continue
for alias in options.get("aliases", []):
# Replace each alias with the correct name of the parameter
if alias in indict and alias is not param:
outdict[param] = indict[alias]
del indict[alias]
del outdict[alias]
continue
if param not in outdict:
if "default" in options:
......@@ -154,10 +152,9 @@ def parse_params(indict, *specs):
logger.debug("Error parsing: %s", wrong_params)
message = Error(
status=400,
message="Missing or invalid parameters",
message='Missing or invalid parameters',