Merge branch 'master' of gitlab.com:LoveIsGrief/muwire into new-icon

Introduce persister that uses a directory structure

.dockerignore

@@ -0,0 +1,12 @@
+# Dot directories
+.gradle/
+.idea/
+.git/
+
+# Build directories
+build/
+**/build/
+
+# We execute COPY . .
+# Modifying these files would unnecessarily invalidate the build context
+Dockerfile

.gitignore

@@ -4,3 +4,5 @@
 .gradle
 .project
 .classpath
+**/*.rej
+**/*.orig

.tx/config

@@ -0,0 +1,9 @@
+[main]
+host = https://www.transifex.com
+lang_map = he: iw, id: in, ru_RU: ru, sv_SE: sv, tr_TR: tr, uk_UA: uk, yi: ji, zh_CN: zh
+
+[I2P.MuWire]
+file_filter = webui/locale/messages_<lang>.po
+source_file = webui/locale/messages_en.po
+source_lang = en
+minimum_perc = 10

Dockerfile

@@ -0,0 +1,64 @@
+FROM jlesage/baseimage-gui:alpine-3.10-glibc
+
+# Docker image version is provided via build arg.
+ARG DOCKER_IMAGE_VERSION=unknown
+
+# JDK version
+ARG JDK=11
+
+# Important directories
+ARG TMP_DIR=/muwire-tmp
+ENV APP_HOME=/muwire
+
+# Define working directory.
+WORKDIR $TMP_DIR
+
+# Put sources into dir
+COPY . .
+
+# Install final dependencies
+RUN add-pkg openjdk${JDK}-jre
+
+# Build and untar in future distribution dir
+RUN add-pkg --virtual openjdk${JDK}-jdk \
+        && ./gradlew --no-daemon clean assemble \
+        && mkdir -p ${APP_HOME} \
+        # Extract to ${APP_HOME and ignore the first dir
+        # First dir in tar is the "MuWire-<version>"
+        && tar -C ${APP_HOME} --strip 1 -xvf gui/build/distributions/MuWire*.tar \
+        # Cleanup
+        && rm -rf "${TMP_DIR}" /root/.gradle /root/.java \
+        && del-pkg openjdk${JDK}-jdk
+
+WORKDIR ${APP_HOME}
+
+# Maximize only the main/initial window.
+RUN \
+    sed-patch 's/<application type="normal">/<application type="normal" title="MuWire">/' \
+        /etc/xdg/openbox/rc.xml
+
+# Generate and install favicons.
+RUN \
+    APP_ICON_URL=https://github.com/zlatinb/muwire/raw/master/gui/griffon-app/resources/MuWire-128x128.png && \
+    install_app_icon.sh "$APP_ICON_URL"
+
+# Add files.
+COPY docker/rootfs/ /
+
+# Set environment variables.
+ENV APP_NAME="MuWire" \
+    S6_KILL_GRACETIME=8000
+
+# Define mountable directories.
+VOLUME ["$APP_HOME/.MuWire"]
+VOLUME ["/incompletes"]
+VOLUME ["/output"]
+
+
+# Metadata.
+LABEL \
+      org.label-schema.name="muwire" \
+      org.label-schema.description="Docker container for MuWire" \
+      org.label-schema.version="$DOCKER_IMAGE_VERSION" \
+      org.label-schema.vcs-url="https://github.com/zlatinb/muwire" \
+      org.label-schema.schema-version="1.0"

README.md

@@ -2,13 +2,13 @@
 
 MuWire is an easy to use file-sharing program which offers anonymity using [I2P technology](http://geti2p.net).  It works on any platform Java works on, including Windows,MacOS,Linux.
 
-It is inspired by the LimeWire Gnutella client and developped by a former LimeWire developer.
+The current stable release - 0.6.8 is avaiable for download at https://muwire.com.  The latest plugin build and instructions how to install the plugin are available inside I2P at http://muwire.i2p.  
 
-The current stable release - 0.5.5 is avaiable for download at https://muwire.com.  You can find technical documentation in the "doc" folder.
+You can find technical documentation in the [doc] folder.  Also check out the [Wiki] for various other documentation.
 
-### Building
+## Building
 
-You need JDK 8 or newer.  After installing that and setting up the appropriate paths, just type
+You need JDK 9 or newer.  After installing that and setting up the appropriate paths, just type
 
 ```
 ./gradlew clean assemble
@@ -19,23 +19,37 @@ If you want to run the unit tests, type
 ./gradlew clean build
 ```
 
-If you want to build binary bundles that do not depend on Java or I2P, see the https://github.com/zlatinb/muwire-pkg project
+If you want to build binary bundles that do not depend on Java or I2P, see the [muwire-pkg] project
 
-### Running the GUI
+## Running the GUI
 
-After you build the application, look inside `gui/build/distributions`.  Untar/unzip one of the `shadow` files and then run the jar contained inside by typing `java -jar gui-x.y.z.jar` in a terminal or command prompt.
+Type
+```
+./gradlew gui:run
+```
 
 If you have an I2P router running on the same machine that is all you need to do.  If you use a custom I2CP host and port, create a file `i2p.properties` and put `i2cp.tcp.host=<host>` and `i2cp.tcp.port=<port>` in there.  On Windows that file should go into `%HOME%\AppData\Roaming\MuWire`, on Mac into `$HOME/Library/Application Support/MuWire` and on Linux `$HOME/.MuWire`
 
 [Default I2CP port]\: `7654`
 
-### Running the CLI
+## Running the CLI
 
-Look inside `cli-lanterna/build/distributions`.  Untar/unzip one of the `shadow` files and then run the jar contained inside by typing `java -jar cli-lanterna-x.y.z.jar` in a terminal.  The CLI will ask you about the router host and port on startup, no need to edit any files.  However, the CLI does not have an options window yet, so if you need to change any options you will need to edit the configuration files.  The CLI options are documented here https://github.com/zlatinb/muwire/wiki/CLI-Configuration-Options
+Look inside `cli-lanterna/build/distributions`.  Untar/unzip one of the `shadow` files and then run the jar contained inside by typing `java -jar cli-lanterna-x.y.z-all.jar` in a terminal.  The CLI will ask you about the router host and port on startup, no need to edit any files.  However, the CLI does not have an options window yet, so if you need to change any options you will need to edit the configuration files.  The CLI options are documented here [cli options]
 
 The CLI is under active development and doesn't have all the features of the GUI.
 
-### GPG Fingerprint
+## Running the Web UI / Plugin
+
+There is a Web-based UI under development.  It is intended to be run as a plugin to the Java I2P router.  Instructions how to build it are available at the wiki [Plugin] page.
+
+## Docker
+
+MuWire is available as a Docker image.  For more information see the [Docker] page.
+
+## Translations
+If you want to help translate MuWire, instructions are on the wiki https://github.com/zlatinb/muwire/wiki/Translate
+
+## GPG Fingerprint
 
 ```
 471B 9FD4 5517 A5ED 101F  C57D A728 3207 2D52 5E41
@@ -45,3 +59,11 @@ You can find the full key at https://keybase.io/zlatinb
 
 
 [Default I2CP port]: https://geti2p.net/en/docs/ports
+[Wiki]: https://github.com/zlatinb/muwire/wiki
+[doc]: https://github.com/zlatinb/muwire/tree/master/doc
+[muwire-pkg]: https://github.com/zlatinb/muwire-pkg 
+[cli options]: https://github.com/zlatinb/muwire/wiki/CLI-Configuration-Options
+[I2P Github]: https://github.com/i2p/i2p.i2p
+[Plugin]: https://github.com/zlatinb/muwire/wiki/Plugin
+[Docker]: https://github.com/zlatinb/muwire/wiki/Docker
+[jlesage/docker-baseimage-gui]: https://github.com/jlesage/docker-baseimage-gui

TODO.md

@@ -1,8 +1,6 @@
 # TODO List
 
-Not in any particular order yet
-
-### Big Items
+### Network
 
 ##### Bloom Filters
 
@@ -12,15 +10,33 @@ This reduces query traffic by not sending last hop queries to peers that definit
 
 This helps with scalability
 
-##### Web UI, REST Interface, etc.
+### Core
 
-Basically any non-gui non-cli user interface
-
-##### Metadata editing and search
-
-To enable parsing of metadata from known file types and the user editing it or adding manual metadata
-
-### Small Items
-
-* Wrapper of some kind for in-place upgrades
+* Metadata parsing and search
 * Automatic adjustment of number of I2P tunnels
+* Persist trust immediately
+* Check if user-selected download and incomplete locations exist and are writeable
+* Enum i18n
+* Ability to share trust list only with trusted users
+* Confidential files visible only to certain users
+* Public Feed feature
+
+### Chat
+* echo "unknown/innappropriate command" in the console
+* break up lines on CR/LF, send multiple messages
+* Style timestamps and persona names
+* enforce # in room names or ignore it
+* auto-create/join channel on server start
+* jump from notification window to room with message
+
+### Swing GUI
+* I2P Status panel - display message when connected to external router
+* Search box - left identation
+
+### Web UI/Plugin
+* HTML 5 media players
+* Minimal dependency (break up groovy-all.jar)
+* Remove versions from jar names
+* Security: POST nonces, CSP headers
+
+ 

build.gradle

@@ -2,8 +2,9 @@ subprojects {
     apply plugin: 'groovy'
 
     dependencies {
-        compile "net.i2p:i2p:${i2pVersion}"
-        compile 'org.codehaus.groovy:groovy-all:2.4.15'
+        compile 'org.codehaus.groovy:groovy:2.4.15'
+        compile 'org.codehaus.groovy:groovy-jsr223:2.4.15'
+        compile 'org.codehaus.groovy:groovy-json:2.4.15'
     }
 
     compileGroovy {

cli-lanterna/build.gradle

@@ -6,7 +6,7 @@ buildscript {
     }
 
     dependencies {
-        classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+        classpath 'com.github.jengelman.gradle.plugins:shadow:5.2.0'
     }
 }
 

cli-lanterna/src/main/groovy/com/muwire/clilanterna/BrowseModel.groovy

@@ -17,7 +17,7 @@ class BrowseModel {
     private final Persona persona
     private final Core core
     private final TextGUIThread guiThread
-    private final TableModel model = new TableModel("Name","Size","Hash","Comment")
+    private final TableModel model = new TableModel("Name","Size","Hash","Comment","Certificates")
     private Map<String, UIResultEvent> rootToResult = new HashMap<>()
     
     private int totalResults
@@ -53,7 +53,7 @@ class BrowseModel {
             String size = DataHelper.formatSize2Decimal(e.size, false) + "B"
             String infoHash = Base64.encode(e.infohash.getRoot())
             String comment = String.valueOf(e.comment != null)
-            model.addRow(e.name, size, infoHash, comment)
+            model.addRow(e.name, size, infoHash, comment, e.certificates)
             rootToResult.put(infoHash, e)
             
             String percentageString = ""
@@ -72,4 +72,27 @@ class BrowseModel {
     void setPercentageLabel(Label percentage) {
         this.percentage = percentage
     }
+    
+    void sort(SortType type) {
+        Comparator<UIResultEvent> chosen
+        switch(type) {
+            case SortType.NAME_ASC : chosen = ResultComparators.NAME_ASC; break
+            case SortType.NAME_DESC : chosen = ResultComparators.NAME_DESC; break
+            case SortType.SIZE_ASC : chosen = ResultComparators.SIZE_ASC; break
+            case SortType.SIZE_DESC : chosen = ResultComparators.SIZE_DESC; break
+        }
+        
+        List<UIResultEvent> l = new ArrayList<>(rootToResult.values())
+        Collections.sort(l, chosen)
+        
+        int rowCount = model.getRowCount()
+        rowCount.times { model.removeRow(0) }
+        
+        l.each { e ->
+            String size = DataHelper.formatSize2Decimal(e.size, false) + "B"
+            String infoHash = Base64.encode(e.infohash.getRoot())
+            String comment = String.valueOf(e.comment != null)
+            model.addRow(e.name, size, infoHash, comment, e.certificates)
+        }
+    }
 }

cli-lanterna/src/main/groovy/com/muwire/clilanterna/BrowseView.groovy

@@ -49,7 +49,7 @@ class BrowseView extends BasicWindow {
         topPanel.addComponent(percentageLabel, layoutData)
         contentPanel.addComponent(topPanel, layoutData)
         
-        table = new Table("Name","Size","Hash","Comment")
+        table = new Table("Name","Size","Hash","Comment","Certificates")
         table.with { 
             setCellSelection(false)
             setTableModel(model.model)
@@ -58,11 +58,17 @@ class BrowseView extends BasicWindow {
         }
         contentPanel.addComponent(table, layoutData)
         
+        Panel buttonsPanel = new Panel()
+        buttonsPanel.setLayoutManager(new GridLayout(2))
+        Button sortButton = new Button("Sort...", {sort()})
         Button closeButton = new Button("Close",{
             model.unregister()
             close()
         })
-        contentPanel.addComponent(closeButton, layoutData)
+        buttonsPanel.addComponent(sortButton, layoutData)
+        buttonsPanel.addComponent(closeButton, layoutData)
+        
+        contentPanel.addComponent(buttonsPanel, layoutData)
         setComponent(contentPanel)
         
     }
@@ -71,19 +77,24 @@ class BrowseView extends BasicWindow {
         int selectedRow = table.getSelectedRow()
         def row = model.model.getRow(selectedRow)
         String infoHash = row[2]
-        boolean comment = Boolean.parseBoolean(row[3])
-        if (comment) {
+        boolean comment = Boolean.parseBoolean(row[3]) 
+        boolean certificates = row[4] > 0
+        if (comment || certificates) {
             Window prompt = new BasicWindow("Download Or View Comment")
             prompt.setHints([Window.Hint.CENTERED])
             Panel contentPanel = new Panel()
-            contentPanel.setLayoutManager(new GridLayout(3))
+            contentPanel.setLayoutManager(new GridLayout(4))
             Button downloadButton = new Button("Download", {download(infoHash)})
             Button viewButton = new Button("View Comment", {viewComment(infoHash)})
+            Button viewCertificate = new Button("View Certificates",{viewCertificates(infoHash)})
             Button closeButton = new Button("Cancel", {prompt.close()})
             
             contentPanel.with { 
                 addComponent(downloadButton, layoutData)
-                addComponent(viewButton, layoutData)
+                if (comment)
+                    addComponent(viewButton, layoutData)
+                if (certificates)
+                    addComponent(viewCertificate, layoutData)
                 addComponent(closeButton, layoutData)
             }
             
@@ -105,7 +116,21 @@ class BrowseView extends BasicWindow {
     
     private void viewComment(String infoHash) {
         UIResultEvent result = model.rootToResult[infoHash]
-        ViewCommentView view = new ViewCommentView(result, terminalSize)
+        ViewCommentView view = new ViewCommentView(result.comment, result.name, terminalSize)
         textGUI.addWindowAndWait(view)
     }
+    
+    private void viewCertificates(String infoHash) {
+        UIResultEvent result = model.rootToResult[infoHash]
+        ViewCertificatesModel model = new ViewCertificatesModel(result, core, textGUI.getGUIThread())
+        ViewCertificatesView view = new ViewCertificatesView(model, textGUI, core, terminalSize)
+        textGUI.addWindowAndWait(view)
+    }
+    
+    private void sort() {
+        SortPrompt prompt = new SortPrompt(textGUI)
+        SortType type = prompt.prompt()
+        if (type != null)
+            model.sort(type)
+    }
 }

cli-lanterna/src/main/groovy/com/muwire/clilanterna/CertificateWrapper.groovy

@@ -0,0 +1,14 @@
+package com.muwire.clilanterna
+
+import com.muwire.core.filecert.Certificate
+
+class CertificateWrapper {
+    private final Certificate certificate
+    CertificateWrapper(Certificate certificate) {
+        this.certificate = certificate
+    }
+    
+    public String toString() {
+        certificate.issuer.getHumanReadableName()
+    }
+}

cli-lanterna/src/main/groovy/com/muwire/clilanterna/ChatConsoleModel.groovy

@@ -0,0 +1,88 @@
+package com.muwire.clilanterna
+
+import com.googlecode.lanterna.gui2.TextBox
+import com.googlecode.lanterna.gui2.TextGUIThread
+import com.muwire.core.Core
+import com.muwire.core.Persona
+import com.muwire.core.chat.ChatConnectionEvent
+import com.muwire.core.chat.ChatLink
+import com.muwire.core.chat.ChatMessageEvent
+import com.muwire.core.chat.UIConnectChatEvent
+
+import net.i2p.data.DataHelper
+
+class ChatConsoleModel {
+    private final Core core
+    private final TextGUIThread guiThread
+    
+    volatile ChatLink link
+    volatile Thread poller
+    volatile boolean running
+    
+    volatile TextBox textBox
+    
+    
+    ChatConsoleModel(Core core, TextGUIThread guiThread) {
+        this.core = core
+        this.guiThread = guiThread
+    }
+    
+    void start() {
+        if (running)
+            return
+        running = true
+        core.chatServer.start()
+        core.eventBus.with { 
+            register(ChatConnectionEvent.class, this)
+            publish(new UIConnectChatEvent(host : core.me))
+        }
+    }
+    
+    void onChatConnectionEvent(ChatConnectionEvent e) {
+        if (e.persona != core.me)
+            return // can't really happen
+        
+        link = e.connection
+        poller = new Thread({eventLoop()} as Runnable)
+        poller.setDaemon(true)
+        poller.start()
+    }
+    
+    void stop() {
+        if (!running)
+            return
+        running = false
+        core.chatServer.stop()
+        poller?.interrupt()
+        link = null
+    }
+    
+    private void eventLoop() {
+        Thread.sleep(1000)
+        while(running) {
+            ChatLink link = this.link
+            if (link == null || !link.isUp()) {
+                Thread.sleep(100)
+                continue
+            }
+                
+            Object event = link.nextEvent()
+            if (event instanceof ChatMessageEvent)
+                handleChatMessage(event)
+            else if (event instanceof Persona)
+                handleLeave(event)
+            else
+                throw new IllegalArgumentException("unknown event type $event")
+        }
+    }
+    
+    private void handleChatMessage(ChatMessageEvent e) {
+        String text = DataHelper.formatTime(e.timestamp)+" <"+e.sender.getHumanReadableName()+ "> ["+
+            e.room+"] "+e.payload
+        guiThread.invokeLater({textBox.addLine(text)})
+    }
+    
+    private void handleLeave(Persona p) {
+        guiThread.invokeLater({textBox.addLine(p.getHumanReadableName()+ " disconnected")})
+    }
+}

cli-lanterna/src/main/groovy/com/muwire/clilanterna/ChatConsoleView.groovy

@@ -0,0 +1,116 @@
+package com.muwire.clilanterna
+
+import com.googlecode.lanterna.TerminalSize
+import com.googlecode.lanterna.gui2.BasicWindow
+import com.googlecode.lanterna.gui2.Button
+import com.googlecode.lanterna.gui2.GridLayout
+import com.googlecode.lanterna.gui2.GridLayout.Alignment
+import com.googlecode.lanterna.gui2.Label
+import com.googlecode.lanterna.gui2.LayoutData
+import com.googlecode.lanterna.gui2.Panel
+import com.googlecode.lanterna.gui2.TextBox
+import com.googlecode.lanterna.gui2.TextGUI
+import com.googlecode.lanterna.gui2.Window
+import com.muwire.core.Core
+import com.muwire.core.chat.ChatCommand
+import com.muwire.core.chat.ChatConnection
+import com.muwire.core.chat.ChatMessageEvent
+import com.muwire.core.chat.ChatServer
+
+import net.i2p.data.DataHelper
+
+class ChatConsoleView extends BasicWindow {
+    private final TextGUI textGUI
+    private final ChatConsoleModel model
+    private final Core core
+    
+    private final LayoutData layoutData = GridLayout.createLayoutData(Alignment.CENTER, Alignment.CENTER, true, false)
+    private final LayoutData layoutDataFill = GridLayout.createLayoutData(Alignment.FILL, Alignment.FILL, true, false)
+    
+    private final TextBox textBox
+    private final TextBox sayField
+    private final TextBox roomField
+    
+    ChatConsoleView(Core core, ChatConsoleModel model, TextGUI textGUI, TerminalSize terminalSize) {
+        super("Chat Server Console")
+        this.core = core
+        this.model = model
+        this.textGUI = textGUI
+        TextBox textBox = model.textBox == null ? new TextBox(terminalSize,"", TextBox.Style.MULTI_LINE) : model.textBox 
+        this.textBox = textBox
+        model.textBox = textBox
+        model.start()
+        TerminalSize textFieldSize = new TerminalSize((terminalSize.getColumns() / 2).toInteger(), 1)
+        this.sayField = new TextBox(textFieldSize,"", TextBox.Style.SINGLE_LINE)
+        this.roomField = new TextBox(textFieldSize,"__CONSOLE__", TextBox.Style.SINGLE_LINE)
+        
+        
+        setHints([Window.Hint.EXPANDED])
+        
+        Panel contentPanel = new Panel()
+        contentPanel.setLayoutManager(new GridLayout(1))
+        contentPanel.addComponent(textBox, layoutData)
+        
+        Panel inputPanel = new Panel()
+        inputPanel.with { 
+            setLayoutManager(new GridLayout(2))
+            addComponent(new Label("Say something here"), layoutData)
+            addComponent(sayField, layoutDataFill)
+            addComponent(new Label("In room:"), layoutData)
+            addComponent(roomField, layoutDataFill)
+        }
+        contentPanel.addComponent(inputPanel, layoutData)
+        
+        Panel bottomPanel = new Panel()
+        bottomPanel.setLayoutManager(new GridLayout(5))
+        
+        Button sayButton = new Button("Say",{say()})
+        Button startButton = new Button("Start Server",{model.start()})
+        Button stopButton = new Button("Stop Server", {model.stop()})
+        Button clearButton = new Button("Clear",{textBox.setText("")})
+        Button closeButton = new Button("Close",{close()})    
+        
+        bottomPanel.with { 
+            addComponent(sayButton, layoutData)
+            addComponent(startButton, layoutData)
+            addComponent(stopButton, layoutData)
+            addComponent(clearButton, layoutData)
+            addComponent(closeButton, layoutData)
+        }    
+        contentPanel.addComponent(bottomPanel, layoutData)
+        setComponent(contentPanel)
+    }
+    
+    private void say() {
+        String command = sayField.getText()
+        sayField.setText("")
+        
+        ChatCommand chatCommand
+        try {
+            chatCommand = new ChatCommand(command)
+        } catch (Exception e) {
+            chatCommand = new ChatCommand("/SAY $command")
+        }
+        command = chatCommand.source
+        
+        String room = roomField.getText()
+        
+        UUID uuid = UUID.randomUUID()
+        long now = System.currentTimeMillis()
+        
+        String toAppend = DataHelper.formatTime(now) + " <" + core.me.getHumanReadableName() + "> [$room] " + command
+        textBox.addLine(toAppend)
+        
+        byte[] sig = ChatConnection.sign(uuid, now, room, command, core.me, core.me, core.spk)
+        
+        def event = new ChatMessageEvent( uuid : uuid,
+            payload : command,
+            sender : core.me,
+            host : core.me,
+            room : room,
+            chatTime : now,
+            sig : sig
+            )
+        core.eventBus.publish(event)
+    }
+}

cli-lanterna/src/main/groovy/com/muwire/clilanterna/CliLanterna.groovy

@@ -32,7 +32,7 @@ import com.muwire.core.UILoadedEvent
 import com.muwire.core.files.AllFilesLoadedEvent
 
 class CliLanterna {
-    private static final String MW_VERSION = "0.5.8"
+    private static final String MW_VERSION = "0.6.8"
     
     private static volatile Core core
     

cli-lanterna/src/main/groovy/com/muwire/clilanterna/FilesModel.groovy

@@ -78,4 +78,32 @@ class FilesModel {
             model.addRow(new SharedFileWrapper(it), DataHelper.formatSize2(size, false)+"B", comment, certified, hits, downloaders)
         }
     }
+    
+    private void sort(SortType type) {
+        Comparator<SharedFile> chosen
+        switch(type) {
+            case SortType.NAME_ASC : chosen = NAME_ASC; break
+            case SortType.NAME_DESC : chosen = NAME_DESC; break
+            case SortType.SIZE_ASC : chosen = SIZE_ASC; break
+            case SortType.SIZE_DESC : chosen = SIZE_DESC; break
+        }
+        
+        Collections.sort(sharedFiles, chosen)
+    }
+    
+    private static final Comparator<SharedFile> NAME_ASC = new Comparator<SharedFile>() {
+        public int compare(SharedFile a, SharedFile b) {
+            a.getFile().getName().compareTo(b.getFile().getName())
+        }
+    }
+    
+    private static final Comparator<SharedFile> NAME_DESC = NAME_ASC.reversed()
+    
+    private static final Comparator<SharedFile> SIZE_ASC = new Comparator<SharedFile>() {
+        public int compare(SharedFile a, SharedFile b) {
+            Long.compare(a.getCachedLength(), b.getCachedLength())
+        }
+    }
+    
+    private static final Comparator<SharedFile> SIZE_DESC = SIZE_ASC.reversed()
 }

cli-lanterna/src/main/groovy/com/muwire/clilanterna/FilesView.groovy

@@ -21,7 +21,6 @@ import com.muwire.core.filecert.UICreateCertificateEvent
 import com.muwire.core.files.DirectoryUnsharedEvent
 import com.muwire.core.files.FileSharedEvent
 import com.muwire.core.files.FileUnsharedEvent
-import com.muwire.core.files.UIPersistFilesEvent
 
 class FilesView extends BasicWindow {
     private final FilesModel model
@@ -51,17 +50,19 @@ class FilesView extends BasicWindow {
         contentPanel.addComponent(table, layoutData)
         
         Panel buttonsPanel = new Panel()
-        buttonsPanel.setLayoutManager(new GridLayout(4))
+        buttonsPanel.setLayoutManager(new GridLayout(5))
         
         Button shareFile = new Button("Share File", {shareFile()})
         Button shareDirectory = new Button("Share Directory", {shareDirectory()})
         Button unshareDirectory = new Button("Unshare Directory",{unshareDirectory()})
+        Button sort = new Button("Sort...",{sort()})
         Button close = new Button("Close", {close()})
         
         buttonsPanel.with { 
             addComponent(shareFile, layoutData)
             addComponent(shareDirectory, layoutData)
             addComponent(unshareDirectory, layoutData)
+            addComponent(sort, layoutData)
             addComponent(close, layoutData)
         }
         
@@ -82,7 +83,6 @@ class FilesView extends BasicWindow {
         
         Button unshareButton = new Button("Unshare", {
             core.eventBus.publish(new FileUnsharedEvent(unsharedFile : sf))
-            core.eventBus.publish(new UIPersistFilesEvent())
             MessageDialog.showMessageDialog(textGUI, "File Unshared", "Unshared "+sf.getFile().getName(), MessageDialogButton.OK)
         } )
         Button addCommentButton = new Button("Add Comment", {
@@ -134,4 +134,11 @@ class FilesView extends BasicWindow {
         core.eventBus.publish(new DirectoryUnsharedEvent(directory : directory))
         MessageDialog.showMessageDialog(textGUI, "Directory Unshared", directory.getName()+" has been unshared", MessageDialogButton.OK)
     }
+    
+    private void sort() {
+        SortPrompt prompt = new SortPrompt(textGUI)
+        SortType type = prompt.prompt()
+        if (type != null)
+            model.sort(type)
+    }
 }

cli-lanterna/src/main/groovy/com/muwire/clilanterna/MainWindowView.groovy

@@ -44,6 +44,7 @@ class MainWindowView extends BasicWindow {
     private final UploadsModel uploadsModel
     private final FilesModel filesModel
     private final TrustModel trustModel
+    private final ChatConsoleModel chatModel
     
     private final Label connectionCount, incoming, outgoing
     private final Label known, failing, hopeless
@@ -63,6 +64,9 @@ class MainWindowView extends BasicWindow {
         uploadsModel = new UploadsModel(textGUI.getGUIThread(), core, props)
         filesModel = new FilesModel(textGUI.getGUIThread(),core)
         trustModel = new TrustModel(textGUI.getGUIThread(), core)
+        chatModel = new ChatConsoleModel(core, textGUI.getGUIThread())
+        if (core.muOptions.startChatServer)
+            core.chatServer.start()
         
         setHints([Window.Hint.EXPANDED])
         Panel contentPanel = new Panel()
@@ -74,7 +78,7 @@ class MainWindowView extends BasicWindow {
         Panel buttonsPanel = new Panel()
         contentPanel.addComponent(buttonsPanel, BorderLayout.Location.TOP)
         
-        GridLayout gridLayout = new GridLayout(7)
+        GridLayout gridLayout = new GridLayout(8)
         buttonsPanel.setLayoutManager(gridLayout)
         
         searchTextBox = new TextBox(new TerminalSize(40, 1))
@@ -83,6 +87,7 @@ class MainWindowView extends BasicWindow {
         Button uploadsButton = new Button("Uploads", {upload()})
         Button filesButton = new Button("Files", { files() })
         Button trustButton = new Button("Trust", {trust()})
+        Button chatButton = new Button("Chat", {chat()})
         Button quitButton = new Button("Quit", {close()})
         
         LayoutData layoutData = GridLayout.createLayoutData(Alignment.CENTER, Alignment.CENTER)
@@ -94,6 +99,7 @@ class MainWindowView extends BasicWindow {
             addComponent(uploadsButton, layoutData)
             addComponent(filesButton, layoutData)
             addComponent(trustButton, layoutData)
+            addComponent(chatButton, layoutData)
             addComponent(quitButton, layoutData)
         }
         
@@ -271,6 +277,10 @@ class MainWindowView extends BasicWindow {
         textGUI.addWindowAndWait(new TrustView(trustModel, textGUI, core, sizeForTables()))
     }
     
+    private void chat() {
+        textGUI.addWindowAndWait(new ChatConsoleView(core, chatModel, textGUI, sizeForTables()))
+    }
+    
     private void refreshStats() {
         int inCon = 0
         int outCon = 0

cli-lanterna/src/main/groovy/com/muwire/clilanterna/ResultComparators.groovy

@@ -0,0 +1,21 @@
+package com.muwire.clilanterna
+
+import com.muwire.core.search.UIResultEvent
+
+class ResultComparators {
+    public static final Comparator<UIResultEvent> NAME_ASC = new Comparator<UIResultEvent>() {
+        public int compare(UIResultEvent a, UIResultEvent b) {
+            a.name.compareTo(b.name)
+        }
+    }
+    
+    public static final Comparator<UIResultEvent> NAME_DESC = NAME_ASC.reversed()
+    
+    public static final Comparator<UIResultEvent> SIZE_ASC = new Comparator<UIResultEvent>() {
+        public int compare(UIResultEvent a, UIResultEvent b) {
+            Long.compare(a.size, b.size)
+        }
+    }
+    
+    public static final Comparator<UIResultEvent> SIZE_DESC = SIZE_ASC.reversed()
+}

cli-lanterna/src/main/groovy/com/muwire/clilanterna/ResultsModel.groovy

@@ -16,7 +16,27 @@ class ResultsModel {
     ResultsModel(UIResultBatchEvent results) {
         this.results = results
         model = new TableModel("Name","Size","Hash","Sources","Comment","Certificates")
-        results.results.each { 
+        updateModel()
+    }
+    
+    void sort(SortType type) {
+        Comparator<UIResultEvent> chosen
+        switch(type) {
+            case SortType.NAME_ASC : chosen = ResultComparators.NAME_ASC; break
+            case SortType.NAME_DESC : chosen = ResultComparators.NAME_DESC; break
+            case SortType.SIZE_ASC : chosen = ResultComparators.SIZE_ASC; break
+            case SortType.SIZE_DESC : chosen = ResultComparators.SIZE_DESC; break
+        }
+
+        Arrays.sort(results.results, chosen)
+        updateModel()
+    }
+    
+    private void updateModel() {
+        int rowCount = model.getRowCount()
+        rowCount.times { model.removeRow(0) }
+        
+        results.results.each {
             String size = DataHelper.formatSize2Decimal(it.size, false) + "B"
             String infoHash = Base64.encode(it.infohash.getRoot())
             String sources = String.valueOf(it.sources.size())

cli-lanterna/src/main/groovy/com/muwire/clilanterna/ResultsView.groovy

@@ -43,9 +43,14 @@ class ResultsView extends BasicWindow {
         table.setTableModel(model.model)
         table.setVisibleRows(terminalSize.getRows())
         contentPanel.addComponent(table, GridLayout.createLayoutData(Alignment.CENTER, Alignment.CENTER, true, false))
-        
+
+        Panel buttonsPanel = new Panel()
+        buttonsPanel.setLayoutManager(new GridLayout(2))
+        Button sortButton = new Button("Sort...",{sort()})
+        buttonsPanel.addComponent(sortButton)        
         Button closeButton = new Button("Close", {close()})
-        contentPanel.addComponent(closeButton, GridLayout.createLayoutData(Alignment.CENTER, Alignment.CENTER, true, false))
+        buttonsPanel.addComponent(closeButton)
+        contentPanel.addComponent(buttonsPanel, GridLayout.createLayoutData(Alignment.CENTER, Alignment.CENTER, true, false))
         
         setComponent(contentPanel)
         closeButton.takeFocus()    
@@ -99,7 +104,7 @@ class ResultsView extends BasicWindow {
     
     private void viewComment(String infohash) {
         UIResultEvent result = model.rootToResult[infohash]
-        ViewCommentView view = new ViewCommentView(result, terminalSize)
+        ViewCommentView view = new ViewCommentView(result.comment, result.name, terminalSize)
         textGUI.addWindowAndWait(view)
     }
     
@@ -109,4 +114,11 @@ class ResultsView extends BasicWindow {
         ViewCertificatesView view = new ViewCertificatesView(model, textGUI, core, terminalSize)
         textGUI.addWindowAndWait(view)
     }
+    
+    private void sort() {
+        SortPrompt prompt = new SortPrompt(textGUI)
+        SortType type = prompt.prompt()
+        if (type != null)
+            model.sort(type)
+    }
 }

cli-lanterna/src/main/groovy/com/muwire/clilanterna/SearchModel.groovy

@@ -7,6 +7,7 @@ import com.muwire.core.search.QueryEvent
 import com.muwire.core.search.SearchEvent
 import com.muwire.core.search.UIResultBatchEvent
 import com.muwire.core.search.UIResultEvent
+import com.muwire.core.util.DataUtil
 
 import net.i2p.crypto.DSAEngine
 import net.i2p.data.Base64
@@ -45,13 +46,16 @@ class SearchModel {
         
         def searchEvent
         byte [] payload
+        UUID uuid = UUID.randomUUID()
+        long timestamp = System.currentTimeMillis()
+        byte [] sig2 = DataUtil.signUUID(uuid, timestamp, core.spk)
         if (hashSearch) {
-            searchEvent = new SearchEvent(searchHash : root, uuid : UUID.randomUUID(), oobInfohash : true, compressedResults : true)
+            searchEvent = new SearchEvent(searchHash : root, uuid : uuid, oobInfohash : true, compressedResults : true)
             payload = root
         } else {
             def nonEmpty = SplitPattern.termify(query)
             payload = String.join(" ", nonEmpty).getBytes(StandardCharsets.UTF_8)
-            searchEvent = new SearchEvent(searchTerms : nonEmpty, uuid : UUID.randomUUID(), oobInfohash: true,
+            searchEvent = new SearchEvent(searchTerms : nonEmpty, uuid : uuid, oobInfohash: true,
             searchComments : core.muOptions.searchComments, compressedResults : true)
         }
         
@@ -61,7 +65,7 @@ class SearchModel {
         
         core.eventBus.publish(new QueryEvent(searchEvent : searchEvent, firstHop : firstHop,
             replyTo: core.me.destination, receivedOn: core.me.destination,
-            originator : core.me, sig: sig.data))
+            originator : core.me, sig: sig.data, queryTime : timestamp, sig2 : sig2))
     }
     
     void unregister() {

cli-lanterna/src/main/groovy/com/muwire/clilanterna/SortPrompt.groovy

@@ -0,0 +1,57 @@
+package com.muwire.clilanterna
+
+import com.googlecode.lanterna.gui2.BasicWindow
+import com.googlecode.lanterna.gui2.Button
+import com.googlecode.lanterna.gui2.GridLayout
+import com.googlecode.lanterna.gui2.GridLayout.Alignment
+import com.googlecode.lanterna.gui2.LayoutData
+import com.googlecode.lanterna.gui2.Panel
+import com.googlecode.lanterna.gui2.TextGUI
+import com.googlecode.lanterna.gui2.Window
+
+class SortPrompt extends BasicWindow {
+    private final TextGUI textGUI
+    private SortType type
+    SortPrompt(TextGUI textGUI) {
+        super("Select what to sort by")
+        this.textGUI = textGUI
+    }
+    
+    SortType prompt() {
+        setHints([Window.Hint.CENTERED])
+        Panel contentPanel = new Panel()
+        contentPanel.setLayoutManager(new GridLayout(5))
+        
+        LayoutData layoutData = GridLayout.createLayoutData(Alignment.CENTER, Alignment.CENTER)
+        
+        Button nameAsc = new Button("Name (ascending)",{
+            type = SortType.NAME_ASC
+            close()  
+        })
+        Button nameDesc = new Button("Name (descending)",{
+            type = SortType.NAME_DESC
+            close()
+        })
+        Button sizeAsc = new Button("Size (ascending)",{
+            type = SortType.SIZE_ASC
+            close()
+        })
+        Button sizeDesc = new Button("Size (descending)",{
+            type = SortType.SIZE_DESC
+            close()
+        })
+        Button close = new Button("Cancel",{close()})
+        
+        contentPanel.with { 
+            addComponent(nameAsc, layoutData)
+            addComponent(nameDesc, layoutData)
+            addComponent(sizeAsc, layoutData)
+            addComponent(sizeDesc, layoutData)
+            addComponent(close, layoutData)
+        }
+        
+        setComponent(contentPanel)
+        textGUI.addWindowAndWait(this)
+        type
+    }
+}

cli-lanterna/src/main/groovy/com/muwire/clilanterna/SortType.java

@@ -0,0 +1,5 @@
+package com.muwire.clilanterna;
+
+public enum SortType {
+    NAME_ASC,NAME_DESC,SIZE_ASC,SIZE_DESC
+}

cli-lanterna/src/main/groovy/com/muwire/clilanterna/TrustEntryWrapper.groovy

@@ -0,0 +1,7 @@
+package com.muwire.clilanterna
+
+import com.muwire.core.trust.TrustService
+
+class TrustEntryWrapper {
+    TrustService.TrustEntry entry
+}

cli-lanterna/src/main/groovy/com/muwire/clilanterna/TrustListModel.groovy

@@ -16,8 +16,8 @@ class TrustListModel {
         this.trustList = trustList
         this.core = core
         
-        trustedTableModel = new TableModel("Trusted User","Your Trust")
-        distrustedTableModel = new TableModel("Distrusted User", "Your Trust")
+        trustedTableModel = new TableModel("Trusted User","Reason","Your Trust")
+        distrustedTableModel = new TableModel("Distrusted User", "Reason", "Your Trust")
         refreshModels()
         
         core.eventBus.register(TrustEvent.class, this)
@@ -36,10 +36,10 @@ class TrustListModel {
         distrustRows.times { distrustedTableModel.removeRow(0) }
         
         trustList.good.each { 
-            trustedTableModel.addRow(new PersonaWrapper(it), core.trustService.getLevel(it.destination))
+            trustedTableModel.addRow(new PersonaWrapper(it.persona),it.reason, core.trustService.getLevel(it.persona.destination))
         }
         trustList.bad.each { 
-            distrustedTableModel.addRow(new PersonaWrapper(it), core.trustService.getLevel(it.destination))
+            distrustedTableModel.addRow(new PersonaWrapper(it.persona),it.reason, core.trustService.getLevel(it.persona.destination))
         }
     }
     

cli-lanterna/src/main/groovy/com/muwire/clilanterna/TrustListView.groovy

@@ -12,6 +12,7 @@ import com.googlecode.lanterna.gui2.TextGUI
 import com.googlecode.lanterna.gui2.Window
 import com.googlecode.lanterna.gui2.dialogs.MessageDialog
 import com.googlecode.lanterna.gui2.dialogs.MessageDialogButton
+import com.googlecode.lanterna.gui2.dialogs.TextInputDialog
 import com.googlecode.lanterna.gui2.table.Table
 import com.muwire.core.Core
 import com.muwire.core.Persona
@@ -48,7 +49,7 @@ class TrustListView extends BasicWindow {
         Panel topPanel = new Panel()
         topPanel.setLayoutManager(new GridLayout(2))
         
-        trusted = new Table("Trusted User","Your Trust")
+        trusted = new Table("Trusted User","Reason","Your Trust")
         trusted.with {
             setCellSelection(false)
             setTableModel(model.trustedTableModel)
@@ -57,7 +58,7 @@ class TrustListView extends BasicWindow {
         trusted.setSelectAction({ actionsForUser(true) })
         topPanel.addComponent(trusted, layoutData)
         
-        distrusted = new Table("Distrusted User", "Your Trust")
+        distrusted = new Table("Distrusted User","Reason", "Your Trust")
         distrusted.with { 
             setCellSelection(false)
             setTableModel(model.distrustedTableModel)
@@ -90,7 +91,8 @@ class TrustListView extends BasicWindow {
         LayoutData layoutData = GridLayout.createLayoutData(Alignment.CENTER, Alignment.CENTER)
         
         Button trustButton = new Button("Trust",{
-            core.eventBus.publish(new TrustEvent(persona : persona, level : TrustLevel.TRUSTED))
+            String reason = TextInputDialog.showDialog(textGUI, "Reason", "Enter reason (optional)", "")
+            core.eventBus.publish(new TrustEvent(persona : persona, level : TrustLevel.TRUSTED, reason : reason))
             MessageDialog.showMessageDialog(textGUI, "Marked Trusted", persona.getHumanReadableName() + "has been marked trusted",
                 MessageDialogButton.OK)
         })
@@ -100,7 +102,8 @@ class TrustListView extends BasicWindow {
                 MessageDialogButton.OK)
         })
         Button distrustButton = new Button("Distrust",{
-            core.eventBus.publish(new TrustEvent(persona : persona, level : TrustLevel.DISTRUSTED))
+            String reason = TextInputDialog.showDialog(textGUI, "Reason", "Enter reason (optional)", "")
+            core.eventBus.publish(new TrustEvent(persona : persona, level : TrustLevel.DISTRUSTED, reason : reason))
             MessageDialog.showMessageDialog(textGUI, "Marked Distrusted", persona.getHumanReadableName() + "has been marked distrusted",
                 MessageDialogButton.OK)
         })

cli-lanterna/src/main/groovy/com/muwire/clilanterna/TrustModel.groovy

@@ -17,8 +17,8 @@ class TrustModel {
         this.guiThread = guiThread
         this.core = core
         
-        modelTrusted = new TableModel("Trusted Users")
-        modelDistrusted = new TableModel("Distrusted Users")
+        modelTrusted = new TableModel("Trusted Users","Reason")
+        modelDistrusted = new TableModel("Distrusted Users","Reason")
         modelSubscriptions = new TableModel("Name","Trusted","Distrusted","Status","Last Updated")
         
         core.eventBus.register(TrustEvent.class, this)
@@ -57,11 +57,11 @@ class TrustModel {
         subsRows.times { modelSubscriptions.removeRow(0) }
         
         core.trustService.good.values().each { 
-            modelTrusted.addRow(new PersonaWrapper(it))
+            modelTrusted.addRow(new PersonaWrapper(it.persona),it.reason)
         }
         
         core.trustService.bad.values().each { 
-            modelDistrusted.addRow(new PersonaWrapper(it))
+            modelDistrusted.addRow(new PersonaWrapper(it.persona),it.reason)
         }
         
         core.trustSubscriber.remoteTrustLists.values().each { 

cli-lanterna/src/main/groovy/com/muwire/clilanterna/TrustView.groovy

@@ -11,6 +11,7 @@ import com.googlecode.lanterna.gui2.Window
 import com.googlecode.lanterna.gui2.dialogs.MessageDialog
 import com.googlecode.lanterna.gui2.dialogs.MessageDialogBuilder
 import com.googlecode.lanterna.gui2.dialogs.MessageDialogButton
+import com.googlecode.lanterna.gui2.dialogs.TextInputDialog
 import com.googlecode.lanterna.gui2.GridLayout.Alignment
 import com.googlecode.lanterna.gui2.Label
 import com.googlecode.lanterna.gui2.table.Table
@@ -44,14 +45,14 @@ class TrustView extends BasicWindow {
         Panel topPanel = new Panel()
         topPanel.setLayoutManager(new GridLayout(2))
         
-        trusted = new Table("Trusted Users")
+        trusted = new Table("Trusted Users","Reason")
         trusted.setCellSelection(false)
         trusted.setSelectAction({trustedActions()})
         trusted.setTableModel(model.modelTrusted)
         trusted.setVisibleRows(tableSize)
         topPanel.addComponent(trusted, layoutData)
         
-        distrusted = new Table("Distrusted users")
+        distrusted = new Table("Distrusted users","Reason")
         distrusted.setCellSelection(false)
         distrusted.setSelectAction({distrustedActions()})
         distrusted.setTableModel(model.modelDistrusted)
@@ -106,7 +107,8 @@ class TrustView extends BasicWindow {
                  MessageDialogButton.OK)  
         })
         Button markDistrusted = new Button("Mark Distrusted", {
-            core.eventBus.publish(new TrustEvent(persona : persona, level : TrustLevel.DISTRUSTED))
+            String reason = TextInputDialog.showDialog(textGUI, "Reason", "Enter reason (optional)", "")
+            core.eventBus.publish(new TrustEvent(persona : persona, level : TrustLevel.DISTRUSTED, reason : reason))
             MessageDialog.showMessageDialog(textGUI, "Marked Distrusted", persona.getHumanReadableName() + "has been marked distrusted",
                  MessageDialogButton.OK)
         })
@@ -122,7 +124,7 @@ class TrustView extends BasicWindow {
     }
     
     private void distrustedActions() {
-        int selectedRow = trusted.getSelectedRow()
+        int selectedRow = distrusted.getSelectedRow()
         def row = model.modelDistrusted.getRow(selectedRow)
         Persona persona = row[0].persona
         
@@ -138,7 +140,8 @@ class TrustView extends BasicWindow {
                  MessageDialogButton.OK)
         })
         Button markDistrusted = new Button("Mark Trusted", {
-            core.eventBus.publish(new TrustEvent(persona : persona, level : TrustLevel.TRUSTED))
+            String reason = TextInputDialog.showDialog(textGUI, "Reason", "Enter reason (optional)", "")
+            core.eventBus.publish(new TrustEvent(persona : persona, level : TrustLevel.TRUSTED, reason : reason))
             MessageDialog.showMessageDialog(textGUI, "Marked Trusted", persona.getHumanReadableName() + "has been marked trusted",
                  MessageDialogButton.OK)
         })

cli-lanterna/src/main/groovy/com/muwire/clilanterna/ViewCertificatesModel.groovy

@@ -17,8 +17,7 @@ class ViewCertificatesModel {
     private final Core core
     private final TextGUIThread guiThread
     
-    private final TableModel model = new TableModel("Issuer","File Name","Timestamp")
-    private final Map<Persona, Set<Certificate>> byIssuer = new HashMap<>()
+    private final TableModel model = new TableModel("Issuer","Trust Status","File Name","Comment","Timestamp")
     
     private int totalCerts
     
@@ -53,14 +52,8 @@ class ViewCertificatesModel {
     void onCertificateFetchedEvent(CertificateFetchedEvent e) {
         guiThread.invokeLater {
             Date date = new Date(e.certificate.timestamp)
-            model.addRow(new PersonaWrapper(e.certificate.issuer), e.certificate.name.name, date)
-            
-            Set<Certificate> set = byIssuer.get(e.certificate.issuer)
-            if (set == null) {
-                set = new HashSet<>()
-                byIssuer.put(e.certificate.issuer, set)
-            }
-            set.add(e.certificate)
+            model.addRow(new CertificateWrapper(e.certificate), core.trustService.getLevel(e.certificate.issuer.destination),
+                e.certificate.name.name, e.certificate.comment != null, date)
             
             String percentageString = ""
             if (totalCerts > 0) {

cli-lanterna/src/main/groovy/com/muwire/clilanterna/ViewCertificatesView.groovy

@@ -49,7 +49,7 @@ class ViewCertificatesView extends BasicWindow {
         topPanel.addComponent(percentageLabel, layoutData)
         contentPanel.addComponent(topPanel, layoutData)
         
-        table = new Table("Issuer","File Name","Timestamp")
+        table = new Table("Issuer","Trust Status","File Name","Comment","Timestamp")
         table.with { 
             setCellSelection(false)
             setTableModel(model.model)
@@ -69,16 +69,21 @@ class ViewCertificatesView extends BasicWindow {
     private void rowSelected() {
         int selectedRow = table.getSelectedRow()
         def row = model.model.getRow(selectedRow)
-        Persona persona = row[0].persona
+        Certificate certificate = row[0].certificate
         
         Window prompt = new BasicWindow("Import Certificate?")
         prompt.setHints([Window.Hint.CENTERED])
         
         Panel contentPanel = new Panel()
-        contentPanel.setLayoutManager(new GridLayout(2))
-        Button importButton = new Button("Import", {importCert(persona)})
+        contentPanel.setLayoutManager(new GridLayout(3))
+        Button importButton = new Button("Import", {importCert(certificate)})
+        
+        Button viewCommentButton = new Button("View Comment", {viewComment(certificate)})
+        
         Button closeButton = new Button("Close", {prompt.close()})
         contentPanel.addComponent(importButton, layoutData)
+        if (certificate.comment != null)
+            contentPanel.addComponent(viewCommentButton, layoutData)
         contentPanel.addComponent(closeButton, layoutData)
         
         prompt.setComponent(contentPanel)
@@ -86,11 +91,13 @@ class ViewCertificatesView extends BasicWindow {
         textGUI.addWindowAndWait(prompt)
     }
     
-    private void importCert(Persona persona) {
-        Set<Certificate> certs = model.byIssuer.get(persona)
-        certs.each { 
-            core.eventBus.publish(new UIImportCertificateEvent(certificate : it))
-        }
+    private void importCert(Certificate certificate) {
+        core.eventBus.publish(new UIImportCertificateEvent(certificate : certificate))
         MessageDialog.showMessageDialog(textGUI, "Certificate(s) Imported", "", MessageDialogButton.OK)
     }
+    
+    private void viewComment(Certificate certificate) {
+        ViewCommentView view = new ViewCommentView(certificate.comment.name, "Certificate Comment", terminalSize)
+        textGUI.addWindowAndWait(view)
+    }
 }

cli-lanterna/src/main/groovy/com/muwire/clilanterna/ViewCommentView.groovy

@@ -19,8 +19,8 @@ class ViewCommentView extends BasicWindow {
     private final TextBox textBox
     private final LayoutData layoutData = GridLayout.createLayoutData(Alignment.CENTER, Alignment.CENTER)
     
-    ViewCommentView(UIResultEvent result, TerminalSize terminalSize) {
-        super("View Comments For "+result.getName())
+    ViewCommentView(String text, String title, TerminalSize terminalSize) {
+        super("View Comments For "+title)
         
         setHints([Window.Hint.CENTERED])
         
@@ -28,7 +28,7 @@ class ViewCommentView extends BasicWindow {
         contentPanel.setLayoutManager(new GridLayout(1))
         
         TerminalSize boxSize = new TerminalSize((terminalSize.getColumns() / 2).toInteger(), (terminalSize.getRows() / 2).toInteger())
-        textBox = new TextBox(boxSize, result.comment, TextBox.Style.MULTI_LINE)
+        textBox = new TextBox(boxSize, text, TextBox.Style.MULTI_LINE)
         contentPanel.addComponent(textBox, layoutData)
         
         Button closeButton = new Button("Close", {close()})

cli/build.gradle

@@ -6,7 +6,7 @@ buildscript {
     }
 
     dependencies {
-        classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+        classpath 'com.github.jengelman.gradle.plugins:shadow:5.2.0'
     }
 }
 

core/build.gradle

@@ -2,6 +2,7 @@ apply plugin : 'application'
 mainClassName = 'com.muwire.core.Core'
 applicationDefaultJvmArgs = ['-Djava.util.logging.config.file=logging.properties']
 dependencies {
+    compile "net.i2p:i2p:${i2pVersion}"
     compile "net.i2p:router:${i2pVersion}"
     compile "net.i2p.client:mstreaming:${i2pVersion}"
     compile "net.i2p.client:streaming:${i2pVersion}"

core/src/main/groovy/com/muwire/core/Core.groovy

@@ -1,8 +1,17 @@
 package com.muwire.core
 
+import com.muwire.core.files.PersisterDoneEvent
+import com.muwire.core.files.PersisterFolderService
+
 import java.nio.charset.StandardCharsets
 import java.util.concurrent.atomic.AtomicBoolean
 
+import com.muwire.core.chat.ChatDisconnectionEvent
+import com.muwire.core.chat.ChatManager
+import com.muwire.core.chat.ChatMessageEvent
+import com.muwire.core.chat.ChatServer
+import com.muwire.core.chat.UIConnectChatEvent
+import com.muwire.core.chat.UIDisconnectChatEvent
 import com.muwire.core.connection.ConnectionAcceptor
 import com.muwire.core.connection.ConnectionEstablisher
 import com.muwire.core.connection.ConnectionEvent
@@ -25,7 +34,6 @@ import com.muwire.core.filecert.UIFetchCertificatesEvent
 import com.muwire.core.filecert.UIImportCertificateEvent
 import com.muwire.core.files.FileDownloadedEvent
 import com.muwire.core.files.FileHashedEvent
-import com.muwire.core.files.FileHashingEvent
 import com.muwire.core.files.FileHasher
 import com.muwire.core.files.FileLoadedEvent
 import com.muwire.core.files.FileManager
@@ -35,7 +43,7 @@ import com.muwire.core.files.HasherService
 import com.muwire.core.files.PersisterService
 import com.muwire.core.files.SideCarFileEvent
 import com.muwire.core.files.UICommentEvent
-import com.muwire.core.files.UIPersistFilesEvent
+
 import com.muwire.core.files.AllFilesLoadedEvent
 import com.muwire.core.files.DirectoryUnsharedEvent
 import com.muwire.core.files.DirectoryWatchedEvent
@@ -68,10 +76,8 @@ import net.i2p.client.I2PClientFactory
 import net.i2p.client.I2PSession
 import net.i2p.client.streaming.I2PSocketManager
 import net.i2p.client.streaming.I2PSocketManagerFactory
-import net.i2p.client.streaming.I2PSocketOptions
 import net.i2p.client.streaming.I2PSocketManager.DisconnectListener
 import net.i2p.crypto.DSAEngine
-import net.i2p.crypto.SigType
 import net.i2p.data.Destination
 import net.i2p.data.PrivateKey
 import net.i2p.data.Signature
@@ -85,13 +91,16 @@ public class Core {
 
     final EventBus eventBus
     final Persona me
+    final String version;
     final File home
     final Properties i2pOptions
     final MuWireSettings muOptions
 
-    private final TrustService trustService
-    private final TrustSubscriber trustSubscriber
+    private final I2PSession i2pSession;
+    final TrustService trustService
+    final TrustSubscriber trustSubscriber
     private final PersisterService persisterService
+    private final PersisterFolderService persisterFolderService
     private final HostCache hostCache
     private final ConnectionManager connectionManager
     private final CacheClient cacheClient
@@ -105,6 +114,8 @@ public class Core {
     final UploadManager uploadManager
     final ContentManager contentManager
     final CertificateManager certificateManager
+    final ChatServer chatServer
+    final ChatManager chatManager
 
     private final Router router
 
@@ -114,26 +125,27 @@ public class Core {
 
     public Core(MuWireSettings props, File home, String myVersion) {
         this.home = home
+        this.version = myVersion
         this.muOptions = props
 
         i2pOptions = new Properties()
-        def i2pOptionsFile = new File(home,"i2p.properties")
+        // Read defaults
+        def defaultI2PFile = getClass()
+                .getClassLoader().getResource("defaults/i2p.properties");
+        defaultI2PFile.withInputStream { i2pOptions.load(it) }
+
+        def i2pOptionsFile = new File(home, "i2p.properties")
         if (i2pOptionsFile.exists()) {
             i2pOptionsFile.withInputStream { i2pOptions.load(it) }
 
             if (!i2pOptions.containsKey("inbound.nickname"))
                 i2pOptions["inbound.nickname"] = "MuWire"
-                if (!i2pOptions.containsKey("outbound.nickname"))
-                    i2pOptions["outbound.nickname"] = "MuWire"
-        } else {
-            i2pOptions["inbound.nickname"] = "MuWire"
-            i2pOptions["outbound.nickname"] = "MuWire"
-            i2pOptions["inbound.length"] = "3"
-            i2pOptions["inbound.quantity"] = "4"
-            i2pOptions["outbound.length"] = "3"
-            i2pOptions["outbound.quantity"] = "4"
-            i2pOptions["i2cp.tcp.host"] = "127.0.0.1"
-            i2pOptions["i2cp.tcp.port"] = "7654"
+            if (!i2pOptions.containsKey("outbound.nickname"))
+                i2pOptions["outbound.nickname"] = "MuWire"
+        }
+        if (!(i2pOptions.hasProperty("i2np.ntcp.port")
+                && i2pOptions.hasProperty("i2np.udp.port")
+        )) {
             Random r = new Random()
             int port = r.nextInt(60000) + 4000
             i2pOptions["i2np.ntcp.port"] = String.valueOf(port)
@@ -142,15 +154,18 @@ public class Core {
         }
 
         if (!props.embeddedRouter) {
-            log.info "Initializing I2P context"
-            I2PAppContext.getGlobalContext().logManager()
-            I2PAppContext.getGlobalContext()._logManager = new MuWireLogManager()
-            router = null
+            if (!(I2PAppContext.getGlobalContext() instanceof RouterContext)) {
+                log.info "Initializing I2P context"
+                I2PAppContext.getGlobalContext().logManager()
+                I2PAppContext.getGlobalContext()._logManager = new MuWireLogManager()
+                router = null
+            }
         } else {
             log.info("launching embedded router")
             Properties routerProps = new Properties()
             routerProps.setProperty("i2p.dir.base", home.getAbsolutePath())
             routerProps.setProperty("i2p.dir.config", home.getAbsolutePath())
+            routerProps.setProperty("geoip.dir", home.getAbsolutePath() + File.separator + "geoip")
             routerProps.setProperty("router.excludePeerCaps", "KLM")
             routerProps.setProperty("i2np.inboundKBytesPerSecond", String.valueOf(props.inBw))
             routerProps.setProperty("i2np.outboundKBytesPerSecond", String.valueOf(props.outBw))
@@ -177,7 +192,6 @@ public class Core {
 
 
         // options like tunnel length and quantity
-        I2PSession i2pSession
         I2PSocketManager socketManager
         keyDat.withInputStream {
             socketManager = new I2PSocketManagerFactory().createManager(it, i2pOptions["i2cp.tcp.host"], i2pOptions["i2cp.tcp.port"].toInteger(), i2pOptions)
@@ -246,7 +260,14 @@ public class Core {
         log.info "initializing persistence service"
         persisterService = new PersisterService(new File(home, "files.json"), eventBus, 60000, fileManager)
         eventBus.register(UILoadedEvent.class, persisterService)
-        eventBus.register(UIPersistFilesEvent.class, persisterService)
+
+        log.info "initializing folder persistence service"
+        persisterFolderService = new PersisterFolderService(this, new File(home, "files"), eventBus)
+        eventBus.register(PersisterDoneEvent.class, persisterFolderService)
+        eventBus.register(FileDownloadedEvent.class, persisterFolderService)
+        eventBus.register(FileLoadedEvent.class, persisterFolderService)
+        eventBus.register(FileHashedEvent.class, persisterFolderService)
+        eventBus.register(FileUnsharedEvent.class, persisterFolderService)
 
         log.info("initializing host cache")
         File hostStorage = new File(home, "hosts.json")
@@ -266,10 +287,13 @@ public class Core {
         log.info("initializing cache client")
         cacheClient = new CacheClient(eventBus,hostCache, connectionManager, i2pSession, props, 10000)
 
+        if (!props.plugin) {
         log.info("initializing update client")
-        updateClient = new UpdateClient(eventBus, i2pSession, myVersion, props, fileManager, me, spk)
-        eventBus.register(FileDownloadedEvent.class, updateClient)
-        eventBus.register(UIResultBatchEvent.class, updateClient)
+            updateClient = new UpdateClient(eventBus, i2pSession, myVersion, props, fileManager, me, spk)
+            eventBus.register(FileDownloadedEvent.class, updateClient)
+            eventBus.register(UIResultBatchEvent.class, updateClient)
+        } else
+            log.info("running as plugin, not initializing update client")
 
         log.info("initializing connector")
         I2PConnector i2pConnector = new I2PConnector(socketManager)
@@ -278,8 +302,16 @@ public class Core {
         CertificateClient certificateClient = new CertificateClient(eventBus, i2pConnector)
         eventBus.register(UIFetchCertificatesEvent.class, certificateClient)
         
+        log.info("initializing chat server")
+        chatServer = new ChatServer(eventBus, props, trustService, me, spk)
+        eventBus.with {
+            register(ChatMessageEvent.class, chatServer)
+            register(ChatDisconnectionEvent.class, chatServer)
+            register(TrustEvent.class, chatServer)
+        }
+        
         log.info "initializing results sender"
-        ResultsSender resultsSender = new ResultsSender(eventBus, i2pConnector, me, props, certificateManager)
+        ResultsSender resultsSender = new ResultsSender(eventBus, i2pConnector, me, props, certificateManager, chatServer)
 
         log.info "initializing search manager"
         SearchManager searchManager = new SearchManager(eventBus, me, resultsSender)
@@ -302,11 +334,21 @@ public class Core {
         log.info("initializing connection establisher")
         connectionEstablisher = new ConnectionEstablisher(eventBus, i2pConnector, props, connectionManager, hostCache)
 
+        
+        log.info("initializing chat manager")
+        chatManager = new ChatManager(eventBus, me, i2pConnector, trustService, props)
+        eventBus.with { 
+            register(UIConnectChatEvent.class, chatManager)
+            register(UIDisconnectChatEvent.class, chatManager)
+            register(ChatMessageEvent.class, chatManager)
+            register(ChatDisconnectionEvent.class, chatManager)
+        }
+        
         log.info("initializing acceptor")
         I2PAcceptor i2pAcceptor = new I2PAcceptor(socketManager)
         connectionAcceptor = new ConnectionAcceptor(eventBus, connectionManager, props,
             i2pAcceptor, hostCache, trustService, searchManager, uploadManager, fileManager, connectionEstablisher,
-            certificateManager)
+            certificateManager, chatServer)
 
         log.info("initializing directory watcher")
         directoryWatcher = new DirectoryWatcher(eventBus, fileManager, home, props)
@@ -331,7 +373,7 @@ public class Core {
         eventBus.register(QueryEvent.class, contentManager)
         
         log.info("initializing browse manager")
-        BrowseManager browseManager = new BrowseManager(i2pConnector, eventBus)
+        BrowseManager browseManager = new BrowseManager(i2pConnector, eventBus, me)
         eventBus.register(UIBrowseEvent.class, browseManager)
         
     }
@@ -346,7 +388,7 @@ public class Core {
         connectionAcceptor.start()
         connectionEstablisher.start()
         hostCache.waitForLoad()
-        updateClient.start()
+        updateClient?.start()
     }
 
     public void shutdown() {
@@ -356,11 +398,19 @@ public class Core {
         }
         log.info("saving settings")
         saveMuSettings()
+        log.info("shutting down host cache")
+        hostCache.stop()
         log.info("shutting down trust subscriber")
         trustSubscriber.stop()
-        log.info("shutting down download manageer")
+        log.info("shutting down trust service")
+        trustService.stop()
+        log.info("shutting down persister service")
+        persisterService.stop()
+        log.info("shutting down persisterFolder service")
+        persisterFolderService.stop()
+        log.info("shutting down download manager")
         downloadManager.shutdown()
-        log.info("shutting down connection acceeptor")
+        log.info("shutting down connection acceptor")
         connectionAcceptor.stop()
         log.info("shutting down connection establisher")
         connectionEstablisher.stop()
@@ -368,12 +418,20 @@ public class Core {
         directoryWatcher.stop()
         log.info("shutting down cache client")
         cacheClient.stop()
+        log.info("shutting down chat server")
+        chatServer.stop()
+        log.info("shutting down chat manager")
+        chatManager.shutdown()
         log.info("shutting down connection manager")
         connectionManager.shutdown()
+        log.info("killing i2p session")
+        i2pSession.destroySession()
         if (router != null) {
             log.info("shutting down embedded router")
             router.shutdown(0)
         }
+        log.info("shutting down event bus");
+        eventBus.shutdown()
         log.info("shutdown complete")
     }
     
@@ -381,6 +439,11 @@ public class Core {
         File f = new File(home, "MuWire.properties")
         f.withPrintWriter("UTF-8", { muOptions.write(it) })
     }
+    
+    public void saveI2PSettings() {
+        File f = new File(home, "i2p.properties")
+        f.withOutputStream { i2pOptions.store(it, "I2P Options") }
+    }
 
     static main(args) {
         def home = System.getProperty("user.home") + File.separator + ".MuWire"
@@ -406,7 +469,7 @@ public class Core {
             }
         }
 
-        Core core = new Core(props, home, "0.5.8")
+        Core core = new Core(props, home, "0.6.8")
         core.startServices()
 
         // ... at the end, sleep or execute script

core/src/main/groovy/com/muwire/core/EventBus.groovy

@@ -2,6 +2,7 @@ package com.muwire.core
 
 import java.util.concurrent.CopyOnWriteArrayList
 import java.util.concurrent.Executor
+import java.util.concurrent.ExecutorService
 import java.util.concurrent.Executors
 import java.util.logging.Level
 
@@ -12,7 +13,7 @@ import groovy.util.logging.Log
 class EventBus {
 
     private Map handlers = new HashMap()
-    private final Executor executor = Executors.newSingleThreadExecutor {r ->
+    private final ExecutorService executor = Executors.newSingleThreadExecutor {r ->
         def rv = new Thread(r)
         rv.setDaemon(true)
         rv.setName("event-bus")
@@ -53,4 +54,8 @@ class EventBus {
         log.info("Unregistering $handler for type $eventType")
         handlers[eventType]?.remove(handler)
     }
+    
+    void shutdown() {
+        executor.shutdownNow()
+    }
 }

core/src/main/groovy/com/muwire/core/MuWireSettings.groovy

@@ -31,12 +31,17 @@ class MuWireSettings {
     boolean shareHiddenFiles
     boolean searchComments
     boolean browseFiles
+    boolean startChatServer
+    int maxChatConnections
+    boolean advertiseChat
+    File chatWelcomeFile
     Set<String> watchedDirectories
     float downloadSequentialRatio
     int hostClearInterval, hostHopelessInterval, hostRejectInterval
     int meshExpiration
     int speedSmoothSeconds
     boolean embeddedRouter
+    boolean plugin
     int inBw, outBw
     Set<String> watchedKeywords
     Set<String> watchedRegexes
@@ -72,6 +77,7 @@ class MuWireSettings {
         hostRejectInterval = Integer.valueOf(props.getProperty("hostRejectInterval", "1"))
         meshExpiration = Integer.valueOf(props.getProperty("meshExpiration","60"))
         embeddedRouter = Boolean.valueOf(props.getProperty("embeddedRouter","false"))
+        plugin = Boolean.valueOf(props.getProperty("plugin","false"))
         inBw = Integer.valueOf(props.getProperty("inBw","256"))
         outBw = Integer.valueOf(props.getProperty("outBw","128"))
         searchComments = Boolean.valueOf(props.getProperty("searchComments","true"))
@@ -79,7 +85,13 @@ class MuWireSettings {
         speedSmoothSeconds = Integer.valueOf(props.getProperty("speedSmoothSeconds","60"))
         totalUploadSlots = Integer.valueOf(props.getProperty("totalUploadSlots","-1"))
         uploadSlotsPerUser = Integer.valueOf(props.getProperty("uploadSlotsPerUser","-1"))
-
+        startChatServer = Boolean.valueOf(props.getProperty("startChatServer","false"))
+        maxChatConnections = Integer.valueOf(props.get("maxChatConnections", "-1"))
+        advertiseChat = Boolean.valueOf(props.getProperty("advertiseChat","true"))
+        String chatWelcomeProp = props.getProperty("chatWelcomeFile")
+        if (chatWelcomeProp != null)
+            chatWelcomeFile = new File(chatWelcomeProp)
+        
         watchedDirectories = DataUtil.readEncodedSet(props, "watchedDirectories")
         watchedKeywords = DataUtil.readEncodedSet(props, "watchedKeywords")
         watchedRegexes = DataUtil.readEncodedSet(props, "watchedRegexes")
@@ -120,6 +132,7 @@ class MuWireSettings {
         props.setProperty("hostRejectInterval", String.valueOf(hostRejectInterval))
         props.setProperty("meshExpiration", String.valueOf(meshExpiration))
         props.setProperty("embeddedRouter", String.valueOf(embeddedRouter))
+        props.setProperty("plugin", String.valueOf(plugin))
         props.setProperty("inBw", String.valueOf(inBw))
         props.setProperty("outBw", String.valueOf(outBw))
         props.setProperty("searchComments", String.valueOf(searchComments))
@@ -127,6 +140,11 @@ class MuWireSettings {
         props.setProperty("speedSmoothSeconds", String.valueOf(speedSmoothSeconds))
         props.setProperty("totalUploadSlots", String.valueOf(totalUploadSlots))
         props.setProperty("uploadSlotsPerUser", String.valueOf(uploadSlotsPerUser))
+        props.setProperty("startChatServer", String.valueOf(startChatServer))
+        props.setProperty("maxChatConnectios", String.valueOf(maxChatConnections))
+        props.setProperty("advertiseChat", String.valueOf(advertiseChat))
+        if (chatWelcomeFile != null)
+            props.setProperty("chatWelcomeFile", chatWelcomeFile.getAbsolutePath())
 
         DataUtil.writeEncodedSet(watchedDirectories, "watchedDirectories", props)
         DataUtil.writeEncodedSet(watchedKeywords, "watchedKeywords", props)

core/src/main/groovy/com/muwire/core/Name.groovy

@@ -1,46 +0,0 @@
-package com.muwire.core
-
-import java.nio.charset.StandardCharsets
-
-/**
- * A name of persona, file or search term
- */
-public class Name {
-    final String name
-
-    Name(String name) {
-        this.name = name
-    }
-
-    Name(InputStream nameStream) throws IOException {
-        DataInputStream dis = new DataInputStream(nameStream)
-        int length = dis.readUnsignedShort()
-        byte [] nameBytes = new byte[length]
-        dis.readFully(nameBytes)
-        this.name = new String(nameBytes, StandardCharsets.UTF_8)
-    }
-
-    public void write(OutputStream out) throws IOException {
-        DataOutputStream dos = new DataOutputStream(out)
-        byte [] bytes = name.getBytes(StandardCharsets.UTF_8)
-        dos.writeShort(bytes.length)
-        dos.write(bytes)
-    }
-
-    public getName() {
-        name
-    }
-
-    @Override
-    public int hashCode() {
-        name.hashCode()
-    }
-
-    @Override
-    public boolean equals(Object o) {
-        if (!(o instanceof Name))
-            return false
-        Name other = (Name)o
-        name.equals(other.name)
-    }
-}

core/src/main/groovy/com/muwire/core/Persona.groovy

@@ -1,94 +0,0 @@
-package com.muwire.core
-
-import net.i2p.crypto.DSAEngine
-import net.i2p.crypto.SigType
-import net.i2p.data.Base64
-import net.i2p.data.Destination
-import net.i2p.data.Signature
-import net.i2p.data.SigningPublicKey
-
-public class Persona {
-    private static final int SIG_LEN = Constants.SIG_TYPE.getSigLen()
-
-    private final byte version
-    private final Name name
-    private final Destination destination
-    private final byte[] sig
-    private volatile String humanReadableName
-    private volatile String base64
-    private volatile byte[] payload
-
-    public Persona(InputStream personaStream) throws IOException, InvalidSignatureException {
-        version = (byte) (personaStream.read() & 0xFF)
-        if (version != Constants.PERSONA_VERSION)
-            throw new IOException("Unknown version "+version)
-
-        name = new Name(personaStream)
-        destination = Destination.create(personaStream)
-        sig = new byte[SIG_LEN]
-        DataInputStream dis = new DataInputStream(personaStream)
-        dis.readFully(sig)
-        if (!verify(version, name, destination, sig))
-            throw new InvalidSignatureException(getHumanReadableName() + " didn't verify")
-    }
-
-    private static boolean verify(byte version, Name name, Destination destination, byte [] sig) {
-        ByteArrayOutputStream baos = new ByteArrayOutputStream()
-        baos.write(version)
-        name.write(baos)
-        destination.writeBytes(baos)
-        byte[] payload = baos.toByteArray()
-        SigningPublicKey spk = destination.getSigningPublicKey()
-        Signature signature = new Signature(Constants.SIG_TYPE, sig)
-        DSAEngine.getInstance().verifySignature(signature, payload, spk)
-    }
-
-    public void write(OutputStream out) throws IOException {
-        if (payload == null) {
-            ByteArrayOutputStream baos = new ByteArrayOutputStream()
-            baos.write(version)
-            name.write(baos)
-            destination.writeBytes(baos)
-            baos.write(sig)
-            payload = baos.toByteArray()
-        }
-        out.write(payload)
-    }
-
-    public String getHumanReadableName() {
-        if (humanReadableName == null)
-            humanReadableName = name.getName() + "@" + destination.toBase32().substring(0,32)
-        humanReadableName
-    }
-
-    public String toBase64() {
-        if (base64 == null) {
-            def baos = new ByteArrayOutputStream()
-            write(baos)
-            base64 = Base64.encode(baos.toByteArray())
-        }
-        base64
-    }
-
-    @Override
-    public int hashCode() {
-        name.hashCode() ^ destination.hashCode()
-    }
-
-    @Override
-    public boolean equals(Object o) {
-        if (!(o instanceof Persona))
-            return false
-        Persona other = (Persona)o
-        name.equals(other.name) && destination.equals(other.destination)
-    }
-
-    public static void main(String []args) {
-        if (args.length != 1) {
-            println "This utility decodes a bas64-encoded persona"
-            System.exit(1)
-        }
-        Persona p = new Persona(new ByteArrayInputStream(Base64.decode(args[0])))
-        println p.getHumanReadableName()
-    }
-}

core/src/main/groovy/com/muwire/core/SplitPattern.groovy

@@ -2,7 +2,7 @@ package com.muwire.core
 
 class SplitPattern {
     
-    public static final String SPLIT_PATTERN = "[\\*\\+\\-,\\.:;\\(\\)=_/\\\\\\!\\\"\\\'\\\$%\\|\\[\\]\\{\\}\\?]";
+    public static final String SPLIT_PATTERN = "[\\*\\+\\-,\\.:;\\(\\)=_/\\\\\\!\\\"\\\'\\\$%\\|\\[\\]\\{\\}\\?\r\n]";
  
     private static final Set<Character> SPLIT_CHARS = new HashSet<>()
     static {

core/src/main/groovy/com/muwire/core/chat/ChatAction.java

@@ -0,0 +1,24 @@
+package com.muwire.core.chat;
+
+enum ChatAction {
+    JOIN(true, false, true, false), 
+    LEAVE(false, false, true, false), 
+    SAY(false, false, true, false), 
+    LIST(true, true, true, false), 
+    HELP(true, true, true, false),
+    INFO(true, true, true, false),
+    JOINED(true, true, false, false),
+    TRUST(true, false, true, true),
+    DISTRUST(true, false, true, true);
+    
+    final boolean console;
+    final boolean stateless;
+    final boolean user;
+    final boolean local;
+    ChatAction(boolean console, boolean stateless, boolean user, boolean local) {
+        this.console = console;
+        this.stateless = stateless;
+        this.user = user;
+        this.local = local;
+    }
+}

core/src/main/groovy/com/muwire/core/chat/ChatClient.groovy

@@ -0,0 +1,141 @@
+package com.muwire.core.chat
+
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.Executor
+import java.util.concurrent.Executors
+
+import com.muwire.core.Constants
+import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
+import com.muwire.core.Persona
+import com.muwire.core.connection.Endpoint
+import com.muwire.core.connection.I2PConnector
+import com.muwire.core.trust.TrustService
+import com.muwire.core.util.DataUtil
+
+import groovy.util.logging.Log
+
+@Log
+class ChatClient implements Closeable {
+    
+    private static final long REJECTION_BACKOFF = 60 * 1000
+    
+    private static final Executor CONNECTOR = Executors.newCachedThreadPool()
+    
+    private final I2PConnector connector
+    private final EventBus eventBus
+    private final Persona host, me
+    private final TrustService trustService
+    private final MuWireSettings settings
+    
+    private ChatConnection connection
+    private boolean connectInProgress
+    private long lastRejectionTime
+    private Thread connectThread
+    
+    ChatClient(I2PConnector connector, EventBus eventBus, Persona host, Persona me, TrustService trustService,
+        MuWireSettings settings) {
+        this.connector = connector
+        this.eventBus = eventBus
+        this.host = host
+        this.me = me
+        this.trustService = trustService
+        this.settings = settings
+    }
+    
+    synchronized void connectIfNeeded() {
+        if (connection != null || connectInProgress || (System.currentTimeMillis() - lastRejectionTime < REJECTION_BACKOFF))
+            return
+        connectInProgress = true
+        CONNECTOR.execute({connect()})
+    }
+    
+    private void connect() {
+        synchronized(this) {
+            if (!connectInProgress)
+                return
+            connectThread = Thread.currentThread()
+        }
+        Endpoint endpoint = null
+        try {
+            eventBus.publish(new ChatConnectionEvent(status : ChatConnectionAttemptStatus.CONNECTING, persona : host))
+            endpoint = connector.connect(host.destination)
+            DataOutputStream dos = new DataOutputStream(endpoint.getOutputStream())
+            DataInputStream dis = new DataInputStream(endpoint.getInputStream())
+            
+            dos.with {
+                write("IRC\r\n".getBytes(StandardCharsets.US_ASCII))
+                write("Version:${Constants.CHAT_VERSION}\r\n".getBytes(StandardCharsets.US_ASCII))
+                write("Persona:${me.toBase64()}\r\n".getBytes(StandardCharsets.US_ASCII))
+                write("\r\n".getBytes(StandardCharsets.US_ASCII))
+                flush()
+            }
+                
+            String codeString = DataUtil.readTillRN(dis)
+            int code = Integer.parseInt(codeString.split(" ")[0])
+            
+            if (code == 429) {
+                eventBus.publish(new ChatConnectionEvent(status : ChatConnectionAttemptStatus.REJECTED, persona : host))
+                try { dos.close() } catch (IOException ignore) {}
+                endpoint.close()
+                synchronized(this) {
+                    lastRejectionTime = System.currentTimeMillis()
+                }
+                return
+            }
+            
+            if (code != 200) 
+                throw new Exception("unknown code $code")
+
+            Map<String,String> headers = DataUtil.readAllHeaders(dis)
+            if (!headers.containsKey('Version'))
+                throw new Exception("Version header missing")
+            
+            int version = Integer.parseInt(headers['Version'])
+            if (version != Constants.CHAT_VERSION)
+                throw new Exception("Unknown chat version $version")
+            
+            synchronized(this) {
+                if (!connectInProgress)
+                    return
+                connection = new ChatConnection(eventBus, endpoint, host, false, trustService, settings)
+                connection.start()
+            }
+            eventBus.publish(new ChatConnectionEvent(status : ChatConnectionAttemptStatus.SUCCESSFUL, persona : host, 
+                connection : connection))
+        } catch (Exception e) {
+            log.log(java.util.logging.Level.WARNING, "connect failed", e)
+            eventBus.publish(new ChatConnectionEvent(status : ChatConnectionAttemptStatus.FAILED, persona : host))
+            if (endpoint != null) {
+                try {endpoint.getOutputStream().close() } catch (IOException ignore) {}
+                endpoint.close()
+            }
+        } finally {
+            synchronized(this) {
+                connectInProgress = false
+                connectThread = null
+            }
+        }
+    }
+    
+    synchronized void disconnected() {
+        connectInProgress = false
+        connection = null
+    }
+    
+    @Override
+    synchronized public void close() {
+        connectInProgress = false
+        connectThread?.interrupt()
+        connection?.close()
+        eventBus.publish(new ChatConnectionEvent(status : ChatConnectionAttemptStatus.DISCONNECTED, persona : host))
+    }
+    
+    synchronized void ping() {
+        connection?.sendPing()
+    }
+    
+    synchronized void sendChat(ChatMessageEvent e) {
+        connection?.sendChat(e)
+    }
+}

core/src/main/groovy/com/muwire/core/chat/ChatCommand.groovy

@@ -0,0 +1,28 @@
+package com.muwire.core.chat
+
+class ChatCommand {
+    private final ChatAction action
+    private final String payload
+    final String source
+    ChatCommand(String source) {
+        if (source.charAt(0) != '/')
+            throw new Exception("command doesn't start with / $source")
+        
+        int position = 1
+        StringBuilder sb = new StringBuilder()
+        while(position < source.length()) {
+            char c = source.charAt(position)
+            if (c == ' ')
+                break
+            sb.append(c)
+            position++    
+        }
+        String command = sb.toString().toUpperCase()
+        action = ChatAction.valueOf(command)
+        if (position < source.length())
+            payload = source.substring(position + 1)
+        else
+            payload = ""
+        this.source = source
+    }
+}

core/src/main/groovy/com/muwire/core/chat/ChatConnection.groovy

@@ -0,0 +1,280 @@
+package com.muwire.core.chat
+
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.BlockingQueue
+import java.util.concurrent.LinkedBlockingQueue
+import java.util.concurrent.atomic.AtomicBoolean
+import java.util.logging.Level
+
+import com.muwire.core.Constants
+import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
+import com.muwire.core.Persona
+import com.muwire.core.connection.Endpoint
+import com.muwire.core.trust.TrustLevel
+import com.muwire.core.trust.TrustService
+
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
+import groovy.util.logging.Log
+import net.i2p.crypto.DSAEngine
+import net.i2p.data.Base64
+import net.i2p.data.Signature
+import net.i2p.data.SigningPrivateKey
+
+@Log
+class ChatConnection implements ChatLink {
+    
+    private static final long PING_INTERVAL = 20000
+    private static final long MAX_CHAT_AGE = 5 * 60 * 1000
+    
+    private final EventBus eventBus
+    private final Endpoint endpoint
+    private final Persona persona
+    private final boolean incoming
+    private final TrustService trustService
+    private final MuWireSettings settings
+    
+    private final AtomicBoolean running = new AtomicBoolean()
+    private final BlockingQueue messages = new LinkedBlockingQueue()
+    private final Thread reader, writer
+    private final LinkedList<Long> timestamps = new LinkedList<>()
+    private final BlockingQueue incomingEvents = new LinkedBlockingQueue()
+    
+    private final DataInputStream dis
+    private final DataOutputStream dos
+    
+    private final JsonSlurper slurper = new JsonSlurper()
+    
+    private volatile long lastPingSentTime
+    
+    ChatConnection(EventBus eventBus, Endpoint endpoint, Persona persona, boolean incoming,
+        TrustService trustService, MuWireSettings settings) {
+        this.eventBus = eventBus
+        this.endpoint = endpoint
+        this.persona = persona
+        this.incoming = incoming
+        this.trustService = trustService
+        this.settings = settings
+        
+        this.dis = new DataInputStream(endpoint.getInputStream())
+        this.dos = new DataOutputStream(endpoint.getOutputStream())
+
+        this.reader = new Thread({readLoop()} as Runnable)
+        this.reader.setName("reader-${persona.getHumanReadableName()}")
+        this.reader.setDaemon(true)
+        
+        this.writer = new Thread({writeLoop()} as Runnable)
+        this.writer.setName("writer-${persona.getHumanReadableName()}")
+        this.writer.setDaemon(true)        
+    }
+    
+    void start() {
+        if (!running.compareAndSet(false, true)) {
+            log.log(Level.WARNING,"${persona.getHumanReadableName()} already running", new Exception())
+            return
+        }
+        reader.start()
+        writer.start()
+    }
+    
+    @Override
+    public boolean isUp() {
+        running.get()
+    }
+    
+    @Override
+    public Persona getPersona() {
+        persona
+    }
+    
+    @Override
+    public void close() {
+        if (!running.compareAndSet(true, false)) {
+            log.log(Level.WARNING,"${persona.getHumanReadableName()} already closed", new Exception())
+            return
+        }
+        log.info("Closing "+persona.getHumanReadableName())
+        reader.interrupt()
+        writer.interrupt()
+        endpoint.close()
+        eventBus.publish(new ChatDisconnectionEvent(persona : persona))
+    }
+    
+    private void readLoop() {
+        try {
+            while(running.get())
+                read()
+        } catch( InterruptedException | SocketTimeoutException ignored) {
+        } catch (Exception e) {
+          log.log(Level.WARNING,"unhandled exception in reader", e)  
+        } finally {
+            try {endpoint.getOutputStream().close()} catch (IOException ignore) {}
+            close()
+        }
+    }
+    
+    private void writeLoop() {
+        try {
+            while(running.get()) {
+                def message = messages.take()
+                write(message)
+            }
+        } catch (InterruptedException ignore) {
+        } catch (Exception e) {
+            log.log(Level.WARNING,"unhandled exception in writer",e)
+        } finally {
+            try {endpoint.getOutputStream().close()} catch (IOException ignore) {}
+            close()
+        }
+    }
+    
+    private void read() {
+        int length = dis.readUnsignedShort()
+        byte [] payload = new byte[length]
+        dis.readFully(payload)
+        def json = slurper.parse(payload)
+        if (json.type == null)
+            throw new Exception("missing json type")
+        switch(json.type) {
+            case "Ping" : break // just ignore
+            case "Chat" : handleChat(json); break
+            case "Leave": handleLeave(json); break
+            default :
+                throw new Exception("unknown json type ${json.type}")
+        }
+    }
+    
+    private void write(Object message) {
+        byte [] payload = JsonOutput.toJson(message).bytes
+        dos.with {
+            writeShort(payload.length)
+            write(payload)
+            flush()
+        }
+    }
+    
+    void sendPing() {
+        long now = System.currentTimeMillis()
+        if (now - lastPingSentTime < PING_INTERVAL)
+            return
+        def ping = [:]
+        ping.type = "Ping"
+        ping.version = 1
+        messages.put(ping)
+        lastPingSentTime = now
+    }
+    
+    private void handleChat(def json) {
+        UUID uuid = UUID.fromString(json.uuid)
+        Persona host = fromString(json.host)
+        Persona sender = fromString(json.sender)
+        long chatTime = json.chatTime
+        String room = json.room
+        String payload = json.payload
+        byte [] sig = Base64.decode(json.sig)
+        
+        if (!verify(uuid,host,sender,chatTime,room,payload,sig)) {
+            log.warning("chat didn't verify")
+            return
+        }
+        if (incoming) {
+            if (sender.destination != endpoint.destination) {
+                log.warning("Sender destination mismatch, dropping message")
+                return
+            }
+        } else {
+            if (host.destination != endpoint.destination) {
+                log.warning("Host destination mismatch, dropping message")
+                return
+            }
+        }
+        if (System.currentTimeMillis() - chatTime > MAX_CHAT_AGE) {
+            log.warning("Chat too old, dropping")
+            return
+        }
+        switch(trustService.getLevel(sender.destination)) {
+            case TrustLevel.TRUSTED : break
+            case TrustLevel.NEUTRAL :
+                if (!settings.allowUntrusted) 
+                    return
+                else
+                    break
+            case TrustLevel.DISTRUSTED :
+                return
+        }
+        def event = new ChatMessageEvent( uuid : uuid, payload : payload, sender : sender,
+            host : host, room : room, chatTime : chatTime, sig : sig)
+        eventBus.publish(event)
+        if (!incoming)
+            incomingEvents.put(event)
+    }
+    
+    private void handleLeave(def json) {
+        Persona leaver = fromString(json.persona)
+        eventBus.publish(new UserDisconnectedEvent(user : leaver, host : persona))
+        incomingEvents.put(leaver)
+    }
+    
+    private static Persona fromString(String base64) {
+        new Persona(new ByteArrayInputStream(Base64.decode(base64)))
+    }
+    
+    private static boolean verify(UUID uuid, Persona host, Persona sender, long chatTime, 
+        String room, String payload, byte []sig) {
+        ByteArrayOutputStream baos = new ByteArrayOutputStream()
+        DataOutputStream daos = new DataOutputStream(baos)
+        daos.write(uuid.toString().bytes)
+        host.write(daos)
+        sender.write(daos)
+        daos.writeLong(chatTime)
+        daos.write(room.getBytes(StandardCharsets.UTF_8))
+        daos.write(payload.getBytes(StandardCharsets.UTF_8))
+        daos.close()
+        byte [] signed = baos.toByteArray()
+        def spk = sender.destination.getSigningPublicKey()
+        def signature = new Signature(Constants.SIG_TYPE, sig)
+        DSAEngine.getInstance().verifySignature(signature, signed, spk)
+    }
+    
+    public static byte[] sign(UUID uuid, long chatTime, String room, String words, Persona sender, Persona host, SigningPrivateKey spk) {
+        ByteArrayOutputStream baos = new ByteArrayOutputStream()
+        DataOutputStream daos = new DataOutputStream(baos)
+        daos.with {
+            write(uuid.toString().bytes)
+            host.write(daos)
+            sender.write(daos)
+            writeLong(chatTime)
+            write(room.getBytes(StandardCharsets.UTF_8))
+            write(words.getBytes(StandardCharsets.UTF_8))
+            close()
+        }
+        byte [] payload = baos.toByteArray()
+        Signature sig = DSAEngine.getInstance().sign(payload, spk)
+        sig.getData()
+    }
+    
+    void sendChat(ChatMessageEvent e) {
+        def chat = [:]
+        chat.type = "Chat"
+        chat.uuid = e.uuid.toString()
+        chat.host = e.host.toBase64()
+        chat.sender = e.sender.toBase64()
+        chat.chatTime = e.chatTime
+        chat.room = e.room
+        chat.payload = e.payload
+        chat.sig = Base64.encode(e.sig)
+        messages.put(chat)
+    }
+    
+    void sendLeave(Persona p) {
+        def leave = [:]
+        leave.type = "Leave"
+        leave.persona = p.toBase64()
+        messages.put(leave)
+    }
+    
+    public Object nextEvent() {
+        incomingEvents.take()
+    }
+}

core/src/main/groovy/com/muwire/core/chat/ChatConnectionAttemptStatus.java

@@ -0,0 +1,5 @@
+package com.muwire.core.chat;
+
+public enum ChatConnectionAttemptStatus {
+    CONNECTING, SUCCESSFUL, REJECTED, FAILED, DISCONNECTED
+}

core/src/main/groovy/com/muwire/core/chat/ChatConnectionEvent.groovy

@@ -0,0 +1,14 @@
+package com.muwire.core.chat
+
+import com.muwire.core.Event
+import com.muwire.core.Persona
+
+class ChatConnectionEvent extends Event {
+    ChatConnectionAttemptStatus status
+    Persona persona
+    ChatLink connection
+    
+    public String toString() {
+        super.toString() + " " + persona.getHumanReadableName() + " " + status.toString()
+    }
+}

core/src/main/groovy/com/muwire/core/chat/ChatDisconnectionEvent.groovy

@@ -0,0 +1,8 @@
+package com.muwire.core.chat
+
+import com.muwire.core.Event
+import com.muwire.core.Persona
+
+class ChatDisconnectionEvent extends Event {
+    Persona persona
+}

core/src/main/groovy/com/muwire/core/chat/ChatLink.java

@@ -0,0 +1,14 @@
+package com.muwire.core.chat;
+
+import java.io.Closeable;
+
+import com.muwire.core.Persona;
+
+public interface ChatLink extends Closeable {
+    public Persona getPersona();
+    public boolean isUp();
+    public void sendChat(ChatMessageEvent e);
+    public void sendLeave(Persona p);
+    public void sendPing();
+    public Object nextEvent() throws InterruptedException;
+}

core/src/main/groovy/com/muwire/core/chat/ChatManager.groovy

@@ -0,0 +1,73 @@
+package com.muwire.core.chat
+
+import java.util.concurrent.ConcurrentHashMap
+
+import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
+import com.muwire.core.Persona
+import com.muwire.core.connection.I2PConnector
+import com.muwire.core.trust.TrustService
+
+class ChatManager {
+    private final EventBus eventBus
+    private final Persona me
+    private final I2PConnector connector
+    private final TrustService trustService
+    private final MuWireSettings settings
+    
+    private final Map<Persona, ChatClient> clients = new ConcurrentHashMap<>()
+    
+    ChatManager(EventBus eventBus, Persona me, I2PConnector connector, TrustService trustService, 
+        MuWireSettings settings) {
+        this.eventBus = eventBus
+        this.me = me
+        this.connector = connector
+        this.trustService = trustService
+        this.settings = settings
+        
+        Timer timer = new Timer("chat-connector", true)
+        timer.schedule({connect()} as TimerTask, 1000, 1000)
+    }
+    
+    void onUIConnectChatEvent(UIConnectChatEvent e) {
+        if (e.host == me) { 
+            eventBus.publish(new ChatConnectionEvent(status : ChatConnectionAttemptStatus.SUCCESSFUL, 
+                persona : me, connection : LocalChatLink.INSTANCE))
+        } else {
+            ChatClient client = new ChatClient(connector, eventBus, e.host, me, trustService, settings)
+            clients.put(e.host, client)
+        }
+    }
+    
+    void onUIDisconnectChatEvent(UIDisconnectChatEvent e) {
+        if (e.host == me)
+            return
+        ChatClient client = clients.remove(e.host)
+        client?.close()
+    }
+    
+    void onChatMessageEvent(ChatMessageEvent e) {
+        if (e.host == me) 
+            return
+        if (e.sender != me)
+            return
+        clients[e.host]?.sendChat(e)
+    }
+    
+    void onChatDisconnectionEvent(ChatDisconnectionEvent e) {
+        clients[e.persona]?.disconnected()
+    }
+    
+    private void connect() {
+        clients.each { k, v -> 
+            v.connectIfNeeded()
+            v.ping() 
+        }
+    }
+    
+    void shutdown() {
+        clients.each { k, v ->
+            v.close() 
+        }
+    }
+}

core/src/main/groovy/com/muwire/core/chat/ChatMessageEvent.groovy

@@ -0,0 +1,13 @@
+package com.muwire.core.chat
+
+import com.muwire.core.Event
+import com.muwire.core.Persona
+
+class ChatMessageEvent extends Event {
+    UUID uuid
+    String payload
+    Persona sender, host
+    String room
+    long chatTime
+    byte [] sig
+}

core/src/main/groovy/com/muwire/core/chat/ChatServer.groovy

@@ -0,0 +1,330 @@
+package com.muwire.core.chat
+
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.atomic.AtomicBoolean
+import java.util.logging.Level
+import java.util.stream.Collectors
+
+import com.muwire.core.Constants
+import com.muwire.core.EventBus
+import com.muwire.core.MuWireSettings
+import com.muwire.core.Persona
+import com.muwire.core.connection.Endpoint
+import com.muwire.core.trust.TrustEvent
+import com.muwire.core.trust.TrustLevel
+import com.muwire.core.trust.TrustService
+import com.muwire.core.util.DataUtil
+
+import groovy.util.logging.Log
+import net.i2p.data.Base64
+import net.i2p.data.Destination
+import net.i2p.data.SigningPrivateKey
+import net.i2p.util.ConcurrentHashSet
+
+@Log
+class ChatServer {
+    public static final String CONSOLE = "__CONSOLE__"
+    private static final String DEFAULT_WELCOME = "Welcome to my chat server!  Type /HELP for list of available commands"
+    
+    private final EventBus eventBus
+    private final MuWireSettings settings
+    private final TrustService trustService
+    private final Persona me
+    private final SigningPrivateKey spk
+    
+    private final Map<Destination, ChatLink> connections = new ConcurrentHashMap()
+    private final Map<String, Set<Persona>> rooms = new ConcurrentHashMap<>()
+    private final Map<Persona, Set<String>> memberships = new ConcurrentHashMap<>()
+    private final Map<String, Persona> shortNames = new ConcurrentHashMap<>()
+    
+    private final AtomicBoolean running = new AtomicBoolean()
+    
+    ChatServer(EventBus eventBus, MuWireSettings settings, TrustService trustService, Persona me, SigningPrivateKey spk) {
+        this.eventBus = eventBus
+        this.settings = settings
+        this.trustService = trustService
+        this.me = me
+        this.spk = spk
+        
+        Timer timer = new Timer("chat-server-pinger", true)
+        timer.schedule({sendPings()} as TimerTask, 1000, 1000)
+    }
+    
+    public void start() {
+        if (!running.compareAndSet(false, true))
+            return
+        connections.put(me.destination, LocalChatLink.INSTANCE)
+        joinRoom(me, CONSOLE)
+        shortNames.put(me.getHumanReadableName(), me)
+        echo(getWelcome(),me.destination)
+    }
+    
+    private String getWelcome() {
+        String welcome = DEFAULT_WELCOME
+        if (settings.chatWelcomeFile != null)
+            welcome = settings.chatWelcomeFile.text
+        "/SAY $welcome"
+    }
+    
+    private void sendPings() {
+        connections.each { k,v -> 
+            v.sendPing()
+        }
+    }
+    
+    public void handle(Endpoint endpoint) {
+        InputStream is = endpoint.getInputStream()
+        OutputStream os = endpoint.getOutputStream()
+        
+        Map<String, String> headers = DataUtil.readAllHeaders(is)
+        
+        if (!headers.containsKey("Version"))
+            throw new Exception("Version header missing")
+        
+        int version = Integer.parseInt(headers['Version'])
+        if (version != Constants.CHAT_VERSION)
+            throw new Exception("Unknown chat version $version")
+        
+        if (!headers.containsKey('Persona'))
+            throw new Exception("Persona header missing")
+        
+        Persona client = new Persona(new ByteArrayInputStream(Base64.decode(headers['Persona'])))
+        if (client.destination != endpoint.destination)
+            throw new Exception("Client destination mismatch")
+        
+        if (!running.get()) {
+            os.write("400 Chat Not Enabled\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+            os.close()
+            endpoint.close()
+            return
+        }
+        
+        if (connections.containsKey(client.destination) || connections.size() == settings.maxChatConnections) {
+            os.write("429 Rejected\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+            os.close()
+            endpoint.close()
+            return
+        }
+        
+        os.with { 
+            write("200 OK\r\n".getBytes(StandardCharsets.US_ASCII))
+            write("Version:${Constants.CHAT_VERSION}\r\n".getBytes(StandardCharsets.US_ASCII))
+            write("\r\n".getBytes(StandardCharsets.US_ASCII))
+            flush()
+        }
+        
+        ChatConnection connection = new ChatConnection(eventBus, endpoint, client, true, trustService, settings)
+        connections.put(endpoint.destination, connection)
+        joinRoom(client, CONSOLE)
+        shortNames.put(client.getHumanReadableName(), client)
+        connection.start()
+        echo(getWelcome(),connection.endpoint.destination)
+    }
+    
+    void onChatDisconnectionEvent(ChatDisconnectionEvent e) {
+        ChatConnection con = connections.remove(e.persona.destination)
+        if (con == null)
+            return
+            
+        Set<String> rooms = memberships.get(e.persona)
+        if (rooms != null) {
+            rooms.each { 
+                leaveRoom(e.persona, it)
+            }
+        }
+        shortNames.remove(e.persona.getHumanReadableName())
+        connections.each { k, v ->
+            v.sendLeave(e.persona)
+        }
+    }
+    
+    void onTrustEvent(TrustEvent e) {
+        if (e.level == TrustLevel.TRUSTED)
+            return
+        if (settings.allowUntrusted && e.level == TrustLevel.NEUTRAL)
+            return
+        
+        ChatConnection connection = connections.get(e.persona.destination)
+        connection?.close()
+    }
+    
+    private void joinRoom(Persona p, String room) {
+        Set<Persona> existing = rooms.get(room)
+        if (existing == null) {
+            existing = new ConcurrentHashSet<>()
+            rooms.put(room, existing)
+        }
+        existing.add(p)
+        
+        Set<String> membership = memberships.get(p)
+        if (membership == null) {
+            membership = new ConcurrentHashSet<>()
+            memberships.put(p, membership)
+        }
+        membership.add(room)
+    }
+    
+    private void leaveRoom(Persona p, String room) {
+        Set<Persona> existing = rooms.get(room)
+        if (existing == null) {
+            log.warning(p.getHumanReadableName() + " leaving room they hadn't joined")
+            return
+        }
+        existing.remove(p)
+        if (existing.isEmpty())
+            rooms.remove(room)
+            
+        Set<String> membership = memberships.get(p)
+        if (membership == null) {
+            log.warning(p.getHumanReadableName() + " didn't have any memberships")
+            return
+        }
+        membership.remove(room)
+        if (membership.isEmpty())
+            memberships.remove(p)
+    }
+    
+    void onChatMessageEvent(ChatMessageEvent e) {
+        if (e.host != me)
+            return
+        
+        ChatCommand command
+        try {
+            command = new ChatCommand(e.payload)
+        } catch (Exception badCommand) {
+            log.log(Level.WARNING, "bad chat command",badCommand)
+            return
+        }
+
+        if ((command.action.console && e.room != CONSOLE) ||
+            (!command.action.console && e.room == CONSOLE) ||
+            !command.action.user)
+            return
+        
+        if (command.action.local && e.sender != me)
+            return
+            
+        switch(command.action) {
+            case ChatAction.JOIN : processJoin(command.payload, e); break
+            case ChatAction.LEAVE : processLeave(e); break
+            case ChatAction.SAY : processSay(e); break
+            case ChatAction.LIST : processList(e.sender.destination); break
+            case ChatAction.INFO : processInfo(e.sender.destination); break
+            case ChatAction.HELP : processHelp(e.sender.destination); break
+            case ChatAction.TRUST : processTrust(command.payload, TrustLevel.TRUSTED); break
+            case ChatAction.DISTRUST : processTrust(command.payload, TrustLevel.DISTRUSTED); break
+        }
+    }
+    
+    private void processJoin(String room, ChatMessageEvent e) {
+        joinRoom(e.sender, room)
+        rooms[room].each { 
+            if (it == e.sender)
+                return
+            connections[it.destination].sendChat(e)
+        }
+        String payload = rooms[room].stream().filter({it != e.sender}).map({it.toBase64()})
+            .collect(Collectors.joining(","))
+        if (payload.length() == 0) {
+            return
+        }
+        payload = "/JOINED $payload"
+        long now = System.currentTimeMillis()
+        UUID uuid = UUID.randomUUID()
+        byte [] sig = ChatConnection.sign(uuid, now, room, payload, me, me, spk)
+        ChatMessageEvent echo = new ChatMessageEvent(
+            uuid : uuid,
+            payload : payload,
+            sender : me,
+            host : me,
+            room : room,
+            chatTime : now,
+            sig : sig
+            )
+        connections[e.sender.destination].sendChat(echo)
+    }
+    
+    private void processLeave(ChatMessageEvent e) {
+        leaveRoom(e.sender, e.room)
+        rooms.getOrDefault(e.room, []).each { 
+            if (it == e.sender)
+                return
+            connections[it.destination].sendChat(e)
+        }
+    }
+    
+    private void processSay(ChatMessageEvent e) {
+        if (rooms.containsKey(e.room)) {
+            // not a private message
+            rooms[e.room].each { 
+                if (it == e.sender)
+                    return
+                connections[it.destination].sendChat(e)
+            }
+        } else {
+            Persona target = new Persona(new ByteArrayInputStream(Base64.decode(e.room)))
+            connections[target.destination]?.sendChat(e)
+        }
+    }
+    
+    private void processList(Destination d) {
+        String roomList = rooms.keySet().stream().filter({it != CONSOLE}).collect(Collectors.joining("\n"))
+        roomList = "/SAY \nRoom List:\n"+roomList
+        echo(roomList, d)
+    }
+    
+    private void processInfo(Destination d) {
+        String info = "/SAY \nThe address of this server is\n========\n${me.toBase64()}\n========\nCopy/paste the above and share it\n"
+        String connectedUsers = memberships.keySet().stream().map({it.getHumanReadableName()}).collect(Collectors.joining("\n"))
+        info = "${info}\nConnected Users:\n$connectedUsers\n======="
+        echo(info, d)
+    }
+    
+    private void processHelp(Destination d) {
+        String help = """/SAY 
+            Available commands: /JOIN /LEAVE /SAY /LIST /INFO /TRUST /DISTRUST /HELP
+            /JOIN <room name>  - joins a room, or creates one if it does not exist.  You must type this in the console
+            /LEAVE             - leaves a room.  You must type this in the room you want to leave
+            /SAY               - optional, says something in the room you're in
+            /LIST              - lists the existing rooms on this server.  You must type this in the console
+            /INFO              - shows information about this server.  You must type this in the console
+            /TRUST <user>      - marks user as trusted.  This is only available to the server owner
+            /DISTRUST <user>   - marks user as distrusted.  This is only available to the server owner
+            /HELP              - prints this help message
+            """
+        echo(help, d)
+    }
+    
+    private void echo(String payload, Destination d) {
+        log.info "echoing $payload"
+        UUID uuid = UUID.randomUUID()
+        long now = System.currentTimeMillis()
+        byte [] sig = ChatConnection.sign(uuid, now, CONSOLE, payload, me, me, spk)
+        ChatMessageEvent echo = new ChatMessageEvent(
+            uuid : uuid,
+            payload : payload,
+            sender : me,
+            host : me,
+            room : CONSOLE,
+            chatTime : now,
+            sig : sig
+            )
+        connections[d]?.sendChat(echo)
+    }
+    
+    private void processTrust(String shortName, TrustLevel level) {
+        Persona p = shortNames.get(shortName)
+        if (p == null)
+            return
+        eventBus.publish(new TrustEvent(persona : p, level : level))
+    }
+    
+    void stop() {
+        if (running.compareAndSet(true, false)) {
+            connections.each { k, v ->
+                v.close()
+            }
+        }
+    }
+}

core/src/main/groovy/com/muwire/core/chat/LocalChatLink.groovy

@@ -0,0 +1,49 @@
+package com.muwire.core.chat
+
+import java.util.concurrent.BlockingQueue
+import java.util.concurrent.LinkedBlockingQueue
+
+import com.muwire.core.Persona
+
+import groovy.util.logging.Log
+
+@Log
+class LocalChatLink implements ChatLink {
+    
+    public static final LocalChatLink INSTANCE = new LocalChatLink()
+    
+    private final BlockingQueue messages = new LinkedBlockingQueue()
+
+    private LocalChatLink() {}
+    
+    @Override
+    public void close() throws IOException {
+    }
+
+    @Override
+    public void sendChat(ChatMessageEvent e) {
+        messages.put(e)
+    }
+
+    @Override
+    public void sendLeave(Persona p) {
+        messages.put(p)
+    }
+
+    @Override
+    public void sendPing() {}
+    
+    @Override
+    public Object nextEvent() {
+        messages.take()
+    }
+
+    @Override
+    public boolean isUp() {
+        true
+    }
+    
+    public Persona getPersona() {
+        null
+    }
+}

core/src/main/groovy/com/muwire/core/chat/UIConnectChatEvent.groovy

@@ -0,0 +1,8 @@
+package com.muwire.core.chat
+
+import com.muwire.core.Event
+import com.muwire.core.Persona
+
+class UIConnectChatEvent extends Event {
+    Persona host
+}

core/src/main/groovy/com/muwire/core/chat/UIDisconnectChatEvent.groovy

@@ -0,0 +1,8 @@
+package com.muwire.core.chat
+
+import com.muwire.core.Event
+import com.muwire.core.Persona
+
+class UIDisconnectChatEvent extends Event {
+    Persona host
+}

core/src/main/groovy/com/muwire/core/chat/UserDisconnectedEvent.groovy

@@ -0,0 +1,9 @@
+package com.muwire.core.chat
+
+import com.muwire.core.Event
+import com.muwire.core.Persona
+
+class UserDisconnectedEvent extends Event {
+    Persona user
+    Persona host
+}

core/src/main/groovy/com/muwire/core/connection/Connection.groovy

@@ -153,6 +153,10 @@ abstract class Connection implements Closeable {
             query.originator = e.originator.toBase64()
         if (e.sig != null)
             query.sig = Base64.encode(e.sig)
+        if (e.queryTime > 0)
+            query.queryTime = e.queryTime
+        if (e.sig2 != null)
+            query.sig2 = Base64.encode(e.sig2)
         messages.put(query)
     }
 
@@ -232,7 +236,6 @@ abstract class Connection implements Closeable {
         if (search.compressedResults != null)
             compressedResults = search.compressedResults
         byte[] sig = null
-        // TODO: make this mandatory at some point
         if (search.sig != null) {
             sig = Base64.decode(search.sig)
             byte [] payload 
@@ -247,21 +250,52 @@ abstract class Connection implements Closeable {
                 return
             } else
                 log.info("query signature verified")
-        } else
+        } else {
             log.info("no signature in query")
+            return
+        }
+        
+        // TODO: make this mandatory at some point
+        byte[] sig2 = null        
+        long queryTime = 0
+        if (search.sig2 != null) {
+            if (search.queryTime == null) {
+                log.info("extended signature but no timestamp")
+                return
+            }
+            sig2 = Base64.decode(search.sig2)
+            queryTime = search.queryTime
+            byte [] payload = (search.uuid + String.valueOf(queryTime)).getBytes(StandardCharsets.US_ASCII)
+            def spk = originator.destination.getSigningPublicKey()
+            def signature = new Signature(Constants.SIG_TYPE, sig2)
+            if (!DSAEngine.getInstance().verifySignature(signature, payload, spk)) {
+                log.info("extended signature didn't match uuid and timestamp")
+                return
+            } else {
+                log.info("extended query signature verified")
+                if (queryTime < System.currentTimeMillis() - Constants.MAX_QUERY_AGE) {
+                    log.info("query too old")
+                    return
+                }
+            }
+        } else
+            log.info("no extended signature in query")
 
         SearchEvent searchEvent = new SearchEvent(searchTerms : search.keywords,
                                             searchHash : infohash,
                                             uuid : uuid,
                                             oobInfohash : oob,
                                             searchComments : searchComments,
-                                            compressedResults : compressedResults)
+                                            compressedResults : compressedResults,
+                                            persona : originator)
         QueryEvent event = new QueryEvent ( searchEvent : searchEvent,
                                             replyTo : replyTo,
                                             originator : originator,
                                             receivedOn : endpoint.destination,
                                             firstHop : search.firstHop,
-                                            sig : sig )
+                                            sig : sig,
+                                            queryTime : queryTime,
+                                            sig2 : sig2 )
         eventBus.publish(event)
 
     }

core/src/main/groovy/com/muwire/core/connection/ConnectionAcceptor.groovy

@@ -15,6 +15,7 @@ import com.muwire.core.EventBus
 import com.muwire.core.InfoHash
 import com.muwire.core.MuWireSettings
 import com.muwire.core.Persona
+import com.muwire.core.chat.ChatServer
 import com.muwire.core.filecert.Certificate
 import com.muwire.core.filecert.CertificateManager
 import com.muwire.core.files.FileManager
@@ -50,6 +51,7 @@ class ConnectionAcceptor {
     final FileManager fileManager
     final ConnectionEstablisher establisher
     final CertificateManager certificateManager
+    final ChatServer chatServer
 
     final ExecutorService acceptorThread
     final ExecutorService handshakerThreads
@@ -61,7 +63,8 @@ class ConnectionAcceptor {
     ConnectionAcceptor(EventBus eventBus, UltrapeerConnectionManager manager,
         MuWireSettings settings, I2PAcceptor acceptor, HostCache hostCache,
         TrustService trustService, SearchManager searchManager, UploadManager uploadManager,
-        FileManager fileManager, ConnectionEstablisher establisher, CertificateManager certificateManager) {
+        FileManager fileManager, ConnectionEstablisher establisher, CertificateManager certificateManager,
+        ChatServer chatServer) {
         this.eventBus = eventBus
         this.manager = manager
         this.settings = settings
@@ -73,6 +76,7 @@ class ConnectionAcceptor {
         this.uploadManager = uploadManager
         this.establisher = establisher
         this.certificateManager = certificateManager
+        this.chatServer = chatServer
 
         acceptorThread = Executors.newSingleThreadExecutor { r ->
             def rv = new Thread(r)
@@ -154,12 +158,17 @@ class ConnectionAcceptor {
                 case (byte)'C':
                     processCERTIFICATES(e)
                     break
+                case (byte)'I':
+                    processIRC(e)
+                    break
                 default:
                     throw new Exception("Invalid read $read")
             }
         } catch (Exception ex) {
             log.log(Level.WARNING, "incoming connection failed",ex)
-            e.getOutputStream().close()
+            try {
+                e.getOutputStream().close()
+            } catch (Exception ignore) {}
             e.close()
             eventBus.publish new ConnectionEvent(endpoint: e, incoming: true, leaf: null, status: ConnectionAttemptStatus.FAILED)
         }
@@ -208,7 +217,9 @@ class ConnectionAcceptor {
                 os.writeShort(json.bytes.length)
                 os.write(json.bytes)
             }
-            e.outputStream.close()
+            try {
+                e.outputStream.close()
+            } catch (Exception ignored) {}
             e.close()
             eventBus.publish(new ConnectionEvent(endpoint: e, incoming: true, leaf: leaf, status: ConnectionAttemptStatus.REJECTED))
         }
@@ -288,23 +299,17 @@ class ConnectionAcceptor {
             if (!searchManager.hasLocalSearch(resultsUUID))
                 throw new UnexpectedResultsException(resultsUUID.toString())
             
-                
             // parse all headers
-            Map<String,String> headers = new HashMap<>()
-            String header
-            while((header = DataUtil.readTillRN(is)) != "" && headers.size() < Constants.MAX_HEADERS) {
-                int colon = header.indexOf(':')
-                if (colon == -1 || colon == header.length() - 1)
-                    throw new IOException("invalid header $header")
-                String key = header.substring(0, colon)
-                String value = header.substring(colon + 1)
-                headers[key] = value.trim()
-            }
+            Map<String,String> headers = DataUtil.readAllHeaders(is);
             
             if (!headers.containsKey("Sender"))
                 throw new IOException("No Sender header")
             if (!headers.containsKey("Count"))
                 throw new IOException("No Count header")
+            
+            boolean chat = false
+            if (headers.containsKey('Chat'))
+                chat = Boolean.parseBoolean(headers['Chat'])
                 
             byte [] personaBytes = Base64.decode(headers['Sender'])
             Persona sender = new Persona(new ByteArrayInputStream(personaBytes))
@@ -323,6 +328,7 @@ class ConnectionAcceptor {
                 dis.readFully(payload)
                 def json = slurper.parse(payload)
                 results[i] = ResultsParser.parse(sender, resultsUUID, json)
+                results[i].chat = chat
             }
             eventBus.publish(new UIResultBatchEvent(uuid: resultsUUID, results: results))
         } catch (IOException bad) {
@@ -340,8 +346,14 @@ class ConnectionAcceptor {
             dis.readFully(rowse)
             if (rowse != "ROWSE\r\n".getBytes(StandardCharsets.US_ASCII))
                 throw new IOException("Invalid BROWSE connection")
-            String header
-            while ((header = DataUtil.readTillRN(dis)) != ""); // ignore headers for now
+                
+            Persona browser = null
+            Map<String,String> headers = DataUtil.readAllHeaders(dis);
+            if (headers.containsKey('Persona')) { 
+                browser = new Persona(new ByteArrayInputStream(Base64.decode(headers['Persona'])))
+                if (browser.destination != e.destination)
+                    throw new IOException("browser persona mismatch")
+            }
 
             OutputStream os = e.getOutputStream()
             if (!settings.browseFiles) {
@@ -357,12 +369,17 @@ class ConnectionAcceptor {
 
             def sharedFiles = fileManager.getSharedFiles().values()
 
-            os.write("Count: ${sharedFiles.size()}\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+            os.write("Count: ${sharedFiles.size()}\r\n".getBytes(StandardCharsets.US_ASCII))
+            
+            boolean chat = chatServer.running.get() && settings.advertiseChat
+            os.write("Chat: ${chat}\r\n".getBytes(StandardCharsets.US_ASCII))
+            
+            os.write("\r\n".getBytes(StandardCharsets.US_ASCII))
 
             DataOutputStream dos = new DataOutputStream(new GZIPOutputStream(os))
             JsonOutput jsonOutput = new JsonOutput()
             sharedFiles.each {
-                it.hit()
+                it.hit(browser, System.currentTimeMillis(), "Browse Host");
                 int certificates = certificateManager.getByInfoHash(it.getInfoHash()).size()
                 def obj = ResultsSender.sharedFileToObj(it, false, certificates)
                 def json = jsonOutput.toJson(obj)
@@ -383,10 +400,10 @@ class ConnectionAcceptor {
             dis.readFully(RUST)
             if (RUST != "RUST\r\n".getBytes(StandardCharsets.US_ASCII))
                 throw new IOException("Invalid TRUST connection")
-            String header
-            while ((header = DataUtil.readTillRN(dis)) != ""); // ignore headers for now
-
-                OutputStream os = e.getOutputStream()
+            
+            Map<String,String> headers = DataUtil.readAllHeaders(dis)
+            
+            OutputStream os = e.getOutputStream()
             if (!settings.allowTrustLists) {
                 os.write("403 Not Allowed\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
                 os.flush()
@@ -394,22 +411,53 @@ class ConnectionAcceptor {
                 return
             }
 
-            os.write("200 OK\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
-            List<Persona> good = new ArrayList<>(trustService.good.values())
-            int size = Math.min(Short.MAX_VALUE * 2, good.size())
-            good = good.subList(0, size)
+            os.write("200 OK\r\n".getBytes(StandardCharsets.US_ASCII))
+            
+            boolean json = headers.containsKey('Json') && Boolean.parseBoolean(headers['Json'])
+            
+            List<TrustService.TrustEntry> good = new ArrayList<>(trustService.good.values())
+            List<TrustService.TrustEntry> bad = new ArrayList<>(trustService.bad.values())
             DataOutputStream dos = new DataOutputStream(os)
-            dos.writeShort(size)
-            good.each {
-                it.write(dos)
-            }
 
-            List<Persona> bad = new ArrayList<>(trustService.bad.values())
-            size = Math.min(Short.MAX_VALUE * 2, bad.size())
-            bad = bad.subList(0, size)
-            dos.writeShort(size)
-            bad.each {
-                it.write(dos)
+            if (!json) {
+                os.write("\r\n".getBytes(StandardCharsets.US_ASCII))
+                int size = Math.min(Short.MAX_VALUE * 2, good.size())
+                good = good.subList(0, size)
+                dos.writeShort(size)
+                good.each {
+                    it.persona.write(dos)
+                }
+
+                size = Math.min(Short.MAX_VALUE * 2, bad.size())
+                bad = bad.subList(0, size)
+                dos.writeShort(size)
+                bad.each {
+                    it.persona.write(dos)
+                }
+            } else {
+                dos.write("Json: true\r\n".getBytes(StandardCharsets.US_ASCII))
+                dos.write("Good:${good.size()}\r\n".getBytes(StandardCharsets.US_ASCII))
+                dos.write("Bad:${bad.size()}\r\n".getBytes(StandardCharsets.US_ASCII))
+                dos.write("\r\n".getBytes(StandardCharsets.US_ASCII))
+                
+                good.each { 
+                    def obj = [:]
+                    obj.persona = it.persona.toBase64()
+                    obj.reason = it.reason
+                    String toJson = JsonOutput.toJson(obj)
+                    byte [] payload = toJson.getBytes(StandardCharsets.US_ASCII)
+                    dos.writeShort(payload.length)
+                    dos.write(payload)   
+                }
+                bad.each {
+                    def obj = [:]
+                    obj.persona = it.persona.toBase64()
+                    obj.reason = it.reason
+                    String toJson = JsonOutput.toJson(obj)
+                    byte [] payload = toJson.getBytes(StandardCharsets.US_ASCII)
+                    dos.writeShort(payload.length)
+                    dos.write(payload)
+                }
             }
 
             dos.flush()
@@ -467,5 +515,14 @@ class ConnectionAcceptor {
             e.close()
         }
     }
+    
+    private void processIRC(Endpoint e) {
+        byte[] IRC = new byte[4]
+        DataInputStream dis = new DataInputStream(e.getInputStream())
+        dis.readFully(IRC)
+        if (IRC != "RC\r\n".getBytes(StandardCharsets.US_ASCII))
+            throw new Exception("Invalid IRC connection")
+        chatServer.handle(e)
+    }
 
 }

core/src/main/groovy/com/muwire/core/download/Downloader.groovy

@@ -92,6 +92,22 @@ public class Downloader {
     public synchronized InfoHash getInfoHash() {
         infoHash
     }
+    
+    public File getFile() {
+        file
+    }
+    
+    public int getNPieces() {
+        nPieces
+    }
+
+    public int getPieceSize() {
+        pieceSize
+    }    
+    
+    public long getLength() {
+        length
+    }
 
     private synchronized void setInfoHash(InfoHash infoHash) {
         this.infoHash = infoHash
@@ -249,6 +265,10 @@ public class Downloader {
         }
         active
     }
+    
+    public int getTotalWorkers() {
+        return activeWorkers.size();
+    }
 
     public void resume() {
         paused = false
@@ -276,6 +296,57 @@ public class Downloader {
         activeWorkers.put(d, newWorker)
         executorService.submit(newWorker)
     }
+    
+    boolean isSequential() {
+        pieces.ratio == 0f
+    }
+    
+    File generatePreview() {
+        int lastCompletePiece = pieces.firstIncomplete() - 1
+        if (lastCompletePiece == -1)
+            return null
+        if (lastCompletePiece < -1)
+            return file
+        long previewableLength = (lastCompletePiece + 1) * ((long)pieceSize)
+        
+        // generate name
+        long now = System.currentTimeMillis()
+        File previewFile
+        File parentFile = file.getParentFile()
+        int lastDot = file.getName().lastIndexOf('.')
+        if (lastDot < 0) 
+            previewFile = new File(parentFile, file.getName() + "." + String.valueOf(now) + ".mwpreview")
+        else {
+            String name = file.getName().substring(0, lastDot)
+            String extension = file.getName().substring(lastDot + 1)
+            String previewName = name + "." + String.valueOf(now) + ".mwpreview."+extension
+            previewFile = new File(parentFile, previewName)
+        }
+        
+        // copy
+        InputStream is = null
+        OutputStream os = null
+        try {
+            is = new BufferedInputStream(new FileInputStream(incompleteFile))
+            os = new BufferedOutputStream(new FileOutputStream(previewFile))
+            byte [] tmp = new byte[0x1 << 13]
+            long totalCopied = 0
+            while(totalCopied < previewableLength) {
+                int read = is.read(tmp, 0, (int)Math.min(tmp.length, previewableLength - totalCopied))
+                if (read < 0)
+                    throw new IOException("EOF?")
+                os.write(tmp, 0, read)
+                totalCopied += read
+            }
+            return previewFile
+        } catch (IOException bad) {
+            log.log(Level.WARNING,"Preview failed",bad)
+            return null
+        } finally {
+            try {is?.close() } catch (IOException ignore) {}
+            try {os?.close() } catch (IOException ignore) {}
+        }
+    }
 
     class DownloadWorker implements Runnable {
         private final Destination destination

core/src/main/groovy/com/muwire/core/download/Pieces.groovy

@@ -108,6 +108,10 @@ class Pieces {
         partials.clear()
     }
     
+    synchronized int firstIncomplete() {
+        done.nextClearBit(0)
+    }
+    
     synchronized void write(PrintWriter writer) {
         for (int i = done.nextSetBit(0); i >= 0; i = done.nextSetBit(i+1)) {
             writer.println(i)

core/src/main/groovy/com/muwire/core/filecert/Certificate.groovy

@@ -10,20 +10,23 @@ import net.i2p.crypto.DSAEngine
 import net.i2p.data.Signature
 import net.i2p.data.SigningPrivateKey
 import net.i2p.data.SigningPublicKey
+import net.i2p.data.Base64
 
 class Certificate {
     private final byte version
     private final InfoHash infoHash
-    private final Name name
-    private final long timestamp
-    private final Persona issuer
+    final Name name, comment
+    final long timestamp
+    final Persona issuer
     private final byte[] sig
     
     private volatile byte [] payload
     
+    private String base64;
+    
     Certificate(InputStream is) {
         version = (byte) (is.read() & 0xFF)
-        if (version != Constants.FILE_CERT_VERSION)
+        if (version > Constants.FILE_CERT_VERSION)
             throw new IOException("Unknown version $version")
         
         DataInputStream dis = new DataInputStream(is)
@@ -35,19 +38,29 @@ class Certificate {
         
         name = new Name(dis)
         
-        issuer = new Persona(is)
+        issuer = new Persona(dis)
+        if (version == 2) {
+            byte present = (byte)(dis.read() & 0xFF)
+            if (present != 0) {
+                comment = new Name(dis)
+            }
+        }
         
         sig = new byte[Constants.SIG_TYPE.getSigLen()]
         dis.readFully(sig)
         
-        if (!verify(version, infoHash, name, timestamp, issuer, sig))
+        if (!verify(version, infoHash, name, timestamp, issuer, comment, sig))
             throw new InvalidSignatureException("certificate for $name.name from ${issuer.getHumanReadableName()} didn't verify")
     }
     
-    Certificate(InfoHash infoHash, String name, long timestamp, Persona issuer, SigningPrivateKey spk) {
+    Certificate(InfoHash infoHash, String name, long timestamp, Persona issuer, String comment, SigningPrivateKey spk) {
         this.version = Constants.FILE_CERT_VERSION
         this.infoHash = infoHash
         this.name = new Name(name)
+        if (comment != null)
+            this.comment = new Name(comment)
+        else
+            this.comment = null
         this.timestamp = timestamp
         this.issuer = issuer
         
@@ -59,6 +72,12 @@ class Certificate {
         daos.write(infoHash.getRoot())
         this.name.write(daos)
         issuer.write(daos)
+        if (this.comment == null) {
+            daos.write((byte) 0)
+        } else {
+            daos.write((byte) 1)
+            this.comment.write(daos)
+        }
         daos.close()
         
         byte[] payload = baos.toByteArray()
@@ -66,7 +85,7 @@ class Certificate {
         this.sig = signature.getData()
     }
     
-    private static boolean verify(byte version, InfoHash infoHash, Name name, long timestamp, Persona issuer, byte[] sig) {
+    private static boolean verify(byte version, InfoHash infoHash, Name name, long timestamp, Persona issuer, Name comment, byte[] sig) {
         ByteArrayOutputStream baos = new ByteArrayOutputStream()
         DataOutputStream daos = new DataOutputStream(baos)
         daos.write(version)
@@ -74,6 +93,14 @@ class Certificate {
         daos.write(infoHash.getRoot())
         name.write(daos)
         issuer.write(daos)
+        if (version == 2) {
+            if (comment == null) {
+                daos.write((byte)0)
+            } else {
+                daos.write((byte)1)
+                comment.write(daos)
+            }
+        }
         daos.close()
         
         byte [] payload = baos.toByteArray()
@@ -91,6 +118,14 @@ class Certificate {
             daos.write(infoHash.getRoot())
             name.write(daos)
             issuer.write(daos)
+            if (version == 2) {
+                if (comment == null)
+                    daos.write((byte) 0)
+                else {
+                    daos.write((byte) 1)
+                    comment.write(daos)
+                }
+            }
             daos.write(sig)
             daos.close()
             
@@ -99,9 +134,18 @@ class Certificate {
         os.write(payload)
     }
     
+    public String toBase64() {
+        if (base64 == null) {
+            ByteArrayOutputStream baos = new ByteArrayOutputStream()
+            write(baos)
+            base64 = Base64.encode(baos.toByteArray())
+        }
+        return base64;
+    }
+    
     @Override
     public int hashCode() {
-        version.hashCode() ^ infoHash.hashCode() ^ timestamp.hashCode() ^ name.hashCode() ^ issuer.hashCode()
+        version.hashCode() ^ infoHash.hashCode() ^ timestamp.hashCode() ^ name.hashCode() ^ issuer.hashCode() ^ Objects.hashCode(comment)
     }
     
     @Override
@@ -114,6 +158,7 @@ class Certificate {
             infoHash == other.infoHash &&
             timestamp == other.timestamp &&
             name == other.name &&
-            issuer == other.issuer
+            issuer == other.issuer &&
+            comment == other.comment
     }
 }

core/src/main/groovy/com/muwire/core/filecert/CertificateClient.groovy

@@ -32,7 +32,8 @@ class CertificateClient {
         fetcherThread.execute({
             Endpoint endpoint = null
             try {
-                eventBus.publish(new CertificateFetchEvent(status : CertificateFetchStatus.CONNECTING))
+                eventBus.publish(new CertificateFetchEvent(status : CertificateFetchStatus.CONNECTING,
+                    user : e.host, infoHash : e.infoHash))
                 endpoint = connector.connect(e.host.destination)
                 
                 String infoHashString = Base64.encode(e.infoHash.getRoot())
@@ -62,7 +63,8 @@ class CertificateClient {
                 int count = Integer.parseInt(headers['Count'])
                 
                 // start pulling the certs
-                eventBus.publish(new CertificateFetchEvent(status : CertificateFetchStatus.FETCHING, count : count))
+                eventBus.publish(new CertificateFetchEvent(status : CertificateFetchStatus.FETCHING, count : count,
+                    user : e.host, infoHash : e.infoHash))
                 
                 DataInputStream dis = new DataInputStream(is)
                 for (int i = 0; i < count; i++) {
@@ -73,14 +75,18 @@ class CertificateClient {
                     try {
                         cert = new Certificate(new ByteArrayInputStream(tmp))
                     } catch (IOException | InvalidSignatureException ignore) {
+                        log.log(Level.WARNING, "certificate creation failed",ignore)
                         continue
                     }
                     if (cert.infoHash == e.infoHash)
-                        eventBus.publish(new CertificateFetchedEvent(certificate : cert))
+                        eventBus.publish(new CertificateFetchedEvent(certificate : cert, user : e.host, infoHash : e.infoHash))
                 }
+                eventBus.publish(new CertificateFetchEvent(status : CertificateFetchStatus.DONE, count : count,
+                    user : e.host, infoHash : e.infoHash))
             } catch (Exception bad) {
                 log.log(Level.WARNING,"Fetching certificates failed", bad)
-                eventBus.publish(new CertificateFetchEvent(status : CertificateFetchStatus.FAILED))
+                eventBus.publish(new CertificateFetchEvent(status : CertificateFetchStatus.FAILED,
+                    user : e.host, infoHash : e.infoHash))
             } finally {
                 endpoint?.close()
             }

core/src/main/groovy/com/muwire/core/filecert/CertificateFetchEvent.groovy

@@ -1,8 +1,12 @@
 package com.muwire.core.filecert
 
 import com.muwire.core.Event
+import com.muwire.core.InfoHash
+import com.muwire.core.Persona
 
 class CertificateFetchEvent extends Event {
     CertificateFetchStatus status
     int count
+    Persona user
+    InfoHash infoHash
 }

core/src/main/groovy/com/muwire/core/filecert/CertificateFetchedEvent.groovy

@@ -1,7 +1,11 @@
 package com.muwire.core.filecert
 
 import com.muwire.core.Event
+import com.muwire.core.InfoHash
+import com.muwire.core.Persona
 
 class CertificateFetchedEvent extends Event {
     Certificate certificate
+    Persona user
+    InfoHash infoHash
 }

core/src/main/groovy/com/muwire/core/filecert/CertificateManager.groovy

@@ -8,6 +8,7 @@ import com.muwire.core.InfoHash
 import com.muwire.core.InvalidSignatureException
 import com.muwire.core.Name
 import com.muwire.core.Persona
+import com.muwire.core.util.DataUtil
 
 import groovy.util.logging.Log
 import net.i2p.data.Base64
@@ -72,7 +73,12 @@ class CertificateManager {
         InfoHash infoHash = e.sharedFile.getInfoHash()
         String name = e.sharedFile.getFile().getName()
         long timestamp = System.currentTimeMillis()
-        Certificate cert = new Certificate(infoHash, name, timestamp, me, spk)
+        
+        String comment = null
+        if (e.sharedFile.getComment() != null)
+            comment = DataUtil.readi18nString(Base64.decode(e.sharedFile.getComment()))
+        
+        Certificate cert = new Certificate(infoHash, name, timestamp, me, comment, spk)
         
         
         if (addToMaps(cert)) {
@@ -90,7 +96,7 @@ class CertificateManager {
 
     private void saveCert(Certificate cert) {
         String infoHashString = Base64.encode(cert.infoHash.getRoot())
-        File certFile = new File(certDir, "${infoHashString}_${cert.issuer.getHumanReadableName()}.mwcert")
+        File certFile = new File(certDir, "${infoHashString}_${cert.issuer.getHumanReadableName()}_${cert.timestamp}.mwcert")
         certFile.withOutputStream { cert.write(it) }
     }
     
@@ -113,7 +119,7 @@ class CertificateManager {
         added
     }
     
-    boolean hasLocalCertificate(InfoHash infoHash) {
+    public boolean hasLocalCertificate(InfoHash infoHash) {
         if (!byInfoHash.containsKey(infoHash))
             return false
         Set<Certificate> set = byInfoHash.get(infoHash)
@@ -124,6 +130,13 @@ class CertificateManager {
         return false
     }
     
+    public boolean isImported(Certificate certificate) {
+        Set<Certificate> forInfoHash = byInfoHash.get(certificate.infoHash)
+        if (forInfoHash == null)
+            return false
+        forInfoHash.contains(certificate)
+    }
+    
     Set<Certificate> getByInfoHash(InfoHash infoHash) {
         Set<Certificate> rv = new HashSet<>()
         if (byInfoHash.containsKey(infoHash))

core/src/main/groovy/com/muwire/core/files/BasePersisterService.groovy

@@ -0,0 +1,109 @@
+package com.muwire.core.files
+
+import com.muwire.core.DownloadedFile
+import com.muwire.core.InfoHash
+import com.muwire.core.Persona
+import com.muwire.core.Service
+import com.muwire.core.SharedFile
+import com.muwire.core.util.DataUtil
+import net.i2p.data.Base64
+import net.i2p.data.Destination
+
+import java.util.stream.Collectors
+
+abstract class BasePersisterService extends Service{
+
+    protected static FileLoadedEvent fromJson(def json) {
+        if (json.file == null || json.length == null || json.infoHash == null || json.hashList == null)
+            throw new IllegalArgumentException()
+        if (!(json.hashList instanceof List))
+            throw new IllegalArgumentException()
+
+        def file = new File(DataUtil.readi18nString(Base64.decode(json.file)))
+        file = file.getCanonicalFile()
+        if (!file.exists() || file.isDirectory())
+            return null
+        long length = Long.valueOf(json.length)
+        if (length != file.length())
+            return null
+
+        List hashList = (List) json.hashList
+        ByteArrayOutputStream baos = new ByteArrayOutputStream()
+        hashList.each {
+            byte [] hash = Base64.decode it.toString()
+            if (hash == null)
+                throw new IllegalArgumentException()
+            baos.write hash
+        }
+        byte[] hashListBytes = baos.toByteArray()
+
+        InfoHash ih = InfoHash.fromHashList(hashListBytes)
+        byte [] root = Base64.decode(json.infoHash.toString())
+        if (root == null)
+            throw new IllegalArgumentException()
+        if (!Arrays.equals(root, ih.getRoot()))
+            return null
+
+        int pieceSize = 0
+        if (json.pieceSize != null)
+            pieceSize = json.pieceSize
+
+        if (json.sources != null) {
+            List sources = (List)json.sources
+            Set<Destination> sourceSet = sources.stream().map({ d -> new Destination(d.toString())}).collect Collectors.toSet()
+            DownloadedFile df = new DownloadedFile(file, ih, pieceSize, sourceSet)
+            df.setComment(json.comment)
+            return new FileLoadedEvent(loadedFile : df)
+        }
+
+
+        SharedFile sf = new SharedFile(file, ih, pieceSize)
+        sf.setComment(json.comment)
+        if (json.downloaders != null)
+            sf.getDownloaders().addAll(json.downloaders)
+        if (json.searchers != null) {
+            json.searchers.each {
+                Persona searcher = null
+                if (it.searcher != null)
+                    searcher = new Persona(new ByteArrayInputStream(Base64.decode(it.searcher)))
+                long timestamp = it.timestamp
+                String query = it.query
+                sf.hit(searcher, timestamp, query)
+            }
+        }
+        return new FileLoadedEvent(loadedFile: sf)
+
+    }
+
+    protected static toJson(SharedFile sf) {
+        def json = [:]
+        json.file = sf.getB64EncodedFileName()
+        json.length = sf.getCachedLength()
+        InfoHash ih = sf.getInfoHash()
+        json.infoHash = sf.getB64EncodedHashRoot()
+        json.pieceSize = sf.getPieceSize()
+        json.hashList = sf.getB64EncodedHashList()
+        json.comment = sf.getComment()
+        json.hits = sf.getHits()
+        json.downloaders = sf.getDownloaders()
+
+        if (!sf.searches.isEmpty()) {
+            Set searchers = new HashSet<>()
+            sf.searches.each {
+                def search = [:]
+                if (it.searcher != null)
+                    search.searcher = it.searcher.toBase64()
+                search.timestamp = it.timestamp
+                search.query = it.query
+                searchers.add(search)
+            }
+            json.searchers = searchers
+        }
+
+        if (sf instanceof DownloadedFile) {
+            json.sources = sf.sources.stream().map( {d -> d.toBase64()}).collect(Collectors.toList())
+        }
+
+        json
+    }
+}

core/src/main/groovy/com/muwire/core/files/FileListCallback.java

@@ -0,0 +1,10 @@
+package com.muwire.core.files;
+
+import java.io.File;
+
+public interface FileListCallback<T> {
+
+    public void onFile(File f, T value);
+    
+    public void onDirectory(File f);
+}

core/src/main/groovy/com/muwire/core/files/FileLoadedEvent.groovy

@@ -6,4 +6,5 @@ import com.muwire.core.SharedFile
 class FileLoadedEvent extends Event {
 
     SharedFile loadedFile
+    String source
 }

core/src/main/groovy/com/muwire/core/files/FileManager.groovy

@@ -24,7 +24,7 @@ class FileManager {
     final Map<String, Set<File>> nameToFiles = new HashMap<>()
     final Map<String, Set<File>> commentToFile = new HashMap<>()
     final SearchIndex index = new SearchIndex()
-    final FileTree negativeTree = new FileTree()
+    final FileTree<Void> negativeTree = new FileTree<>()
     final Set<File> sideCarFiles = new HashSet<>()
 
     FileManager(EventBus eventBus, MuWireSettings settings) {
@@ -32,7 +32,7 @@ class FileManager {
         this.eventBus = eventBus
         
         for (String negative : settings.negativeFileTree) {
-            negativeTree.add(new File(negative))
+            negativeTree.add(new File(negative), null)
         }
     }
     
@@ -88,7 +88,7 @@ class FileManager {
         negativeTree.remove(sf.file)
         String parent = sf.getFile().getParent()
         if (parent != null && settings.watchedDirectories.contains(parent)) {
-            negativeTree.add(sf.file.getParentFile())
+            negativeTree.add(sf.file.getParentFile(),null)
         }
         saveNegativeTree()
 
@@ -128,7 +128,7 @@ class FileManager {
 
         fileToSharedFile.remove(sf.file)
         if (!e.deleted && negativeTree.fileToNode.containsKey(sf.file.getParentFile())) {
-            negativeTree.add(sf.file)
+            negativeTree.add(sf.file,null)
             saveNegativeTree()
         }
 
@@ -143,6 +143,7 @@ class FileManager {
         
         String comment = sf.getComment()
         if (comment != null) {
+            comment = DataUtil.readi18nString(Base64.decode(comment))
             Set<File> existingComment = commentToFile.get(comment)
             if (existingComment != null) {
                 existingComment.remove(sf.getFile())
@@ -198,7 +199,7 @@ class FileManager {
             found = rootToFiles.get new InfoHash(e.searchHash)
             found = filter(found, e.oobInfohash)
             if (found != null && !found.isEmpty()) {
-                found.each { it.hit() }
+                found.each { it.hit(e.persona, e.timestamp, "Hash Search") }
                 re = new ResultsEvent(results: found.asList(), uuid: e.uuid, searchEvent: e)
             }
         } else {
@@ -214,7 +215,7 @@ class FileManager {
             files = filter(sharedFiles, e.oobInfohash)
             
             if (!sharedFiles.isEmpty()) {
-                sharedFiles.each { it.hit() }
+                sharedFiles.each { it.hit(e.persona, e.timestamp, String.join(" ", e.searchTerms)) }
                 re = new ResultsEvent(results: sharedFiles.asList(), uuid: e.uuid, searchEvent: e)
             }
 
@@ -229,7 +230,7 @@ class FileManager {
             return files
         Set<SharedFile> rv = new HashSet<>()
         files.each {
-            if (it.getPieceSize() != 0)
+            if (it != null && it.getPieceSize() != 0)
                 rv.add(it)
         }
         rv

core/src/main/groovy/com/muwire/core/files/FileTree.groovy

@@ -2,12 +2,12 @@ package com.muwire.core.files
 
 import java.util.concurrent.ConcurrentHashMap
 
-class FileTree {
+class FileTree<T> {
     
     private final TreeNode root = new TreeNode()
     private final Map<File, TreeNode> fileToNode = new ConcurrentHashMap<>()
     
-    void add(File file) {
+    synchronized void add(File file, T value) {
         List<File> path = new ArrayList<>()
         path.add(file)
         while (file.getParentFile() != null) {
@@ -29,9 +29,10 @@ class FileTree {
             }
             current = existing
         }
+        current.value = value;
     }
 
-    boolean remove(File file) {
+    synchronized boolean remove(File file) {
         TreeNode node = fileToNode.remove(file)
         if (node == null) {
             return false
@@ -45,13 +46,63 @@ class FileTree {
         true
     }    
     
-    public static class TreeNode {
+    synchronized void traverse(FileTreeCallback<T> callback) {
+        doTraverse(root, callback);
+    }
+
+    synchronized void traverse(File from, FileTreeCallback<T> callback) {
+        if (from == null) {
+            doTraverse(root, callback);
+        } else {
+            TreeNode node = fileToNode.get(from);
+            if (node == null)
+                return
+            doTraverse(node, callback);
+        }
+    }
+        
+    private void doTraverse(TreeNode<T> node, FileTreeCallback<T> callback) {
+        boolean leave = false
+        if (node.file != null) {
+            if (node.file.isFile())
+                callback.onFile(node.file, node.value)
+            else {
+                leave = true
+                callback.onDirectoryEnter(node.file)
+            }
+        }
+        
+        node.children.each { 
+            doTraverse(it, callback)
+        }        
+        
+        if (leave)
+            callback.onDirectoryLeave()
+    }
+    
+    synchronized void list(File parent, FileListCallback<T> callback) {
+        TreeNode<T> node
+        if (parent == null)
+            node = root
+        else 
+            node = fileToNode.get(parent)
+            
+        node.children.each { 
+            if (it.file.isFile())
+                callback.onFile(it.file, it.value)
+            else
+                callback.onDirectory(it.file)
+        }
+    }
+    
+    public static class TreeNode<T> {
         TreeNode parent
         File file
+        T value;
         final Set<TreeNode> children = new HashSet<>()
         
         public int hashCode() {
-            file.hashCode()
+            Objects.hash(file)
         }
         
         public boolean equals(Object o) {

core/src/main/groovy/com/muwire/core/files/FileTreeCallback.java

@@ -0,0 +1,9 @@
+package com.muwire.core.files;
+
+import java.io.File;
+
+public interface FileTreeCallback<T> {
+    public void onDirectoryEnter(File file);
+    public void onDirectoryLeave();
+    public void onFile(File file, T value);
+}

core/src/main/groovy/com/muwire/core/files/PersisterDoneEvent.groovy

@@ -0,0 +1,12 @@
+package com.muwire.core.files
+
+import com.muwire.core.Event
+
+/**
+ * Should be triggered by the old PersisterService
+ * once it has finished reading the old file
+ *
+ * @see PersisterService
+ */
+class PersisterDoneEvent extends Event{
+}

core/src/main/groovy/com/muwire/core/files/PersisterFolderService.groovy

@@ -0,0 +1,150 @@
+package com.muwire.core.files
+
+import com.muwire.core.*
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
+import groovy.util.logging.Log
+
+import java.nio.file.Files
+import java.nio.file.Path
+import java.nio.file.Paths
+import java.util.concurrent.ExecutorService
+import java.util.concurrent.Executors
+import java.util.concurrent.ThreadFactory
+import java.util.logging.Level
+
+/**
+ * A persister that stores information about the files shared using
+ * individual JSON files in directories.
+ *
+ * The absolute path's 32bit hash to the shared file is used
+ * to build the directory and filename.
+ *
+ * This persister only starts working once the old persister has finished loading
+ * @see PersisterFolderService#getJsonPath
+ */
+@Log
+class PersisterFolderService extends BasePersisterService {
+
+    final static int CUT_LENGTH = 6
+
+    private final Core core;
+    final File location
+    final EventBus listener
+    final int interval
+    final Timer timer
+    final ExecutorService persisterExecutor = Executors.newSingleThreadExecutor({ r ->
+        new Thread(r, "file persister")
+    } as ThreadFactory)
+
+    PersisterFolderService(Core core, File location, EventBus listener) {
+        this.core = core;
+        this.location = location
+        this.listener = listener
+        this.interval = interval
+        timer = new Timer("file-folder persister timer", true)
+    }
+
+    void stop() {
+        timer.cancel()
+        persisterExecutor.shutdown()
+    }
+
+    void onPersisterDoneEvent(PersisterDoneEvent persisterDoneEvent) {
+        log.info("Old persister done")
+        load()
+    }
+
+    void onFileHashedEvent(FileHashedEvent hashedEvent) {
+        persistFile(hashedEvent.sharedFile)
+    }
+
+    void onFileDownloadedEvent(FileDownloadedEvent downloadedEvent) {
+        if (core.getMuOptions().getShareDownloadedFiles()) {
+            persistFile(downloadedEvent.downloadedFile)
+        }
+    }
+
+    /**
+     * Get rid of the json of unshared files
+     * @param unsharedEvent
+     */
+    void onFileUnsharedEvent(FileUnsharedEvent unsharedEvent) {
+        def jsonPath = getJsonPath(unsharedEvent.unsharedFile)
+        def jsonFile = jsonPath.toFile()
+        if(jsonFile.isFile()){
+            jsonFile.delete()
+        }
+    }
+    void onFileLoadedEvent(FileLoadedEvent loadedEvent) {
+        if(loadedEvent.source == "PersisterService"){
+            log.info("Migrating persisted file from PersisterService: "
+                    + loadedEvent.loadedFile.file.absolutePath.toString())
+            persistFile(loadedEvent.loadedFile)
+        }
+    }
+
+    void load() {
+        log.fine("Loading...")
+        Thread.currentThread().setPriority(Thread.MIN_PRIORITY)
+
+        if (location.exists() && location.isDirectory()) {
+            try {
+                _load()
+            }
+            catch (IllegalArgumentException e) {
+                log.log(Level.WARNING, "couldn't load files", e)
+            }
+        } else {
+            location.mkdirs()
+            listener.publish(new AllFilesLoadedEvent())
+        }
+        loaded = true
+    }
+
+    /**
+     * Loads every JSON into memory
+     */
+    private void _load() {
+        int loaded = 0
+        def slurper = new JsonSlurper()
+        Files.walk(location.toPath())
+                .filter({ it.fileName.endsWith(".json") })
+                .forEach({
+                    def parsed = slurper.parse it.toFile()
+                    def event = fromJson parsed
+                    if (event == null) return
+
+                    log.fine("loaded file $event.loadedFile.file")
+                    listener.publish event
+                    loaded++
+                    if (loaded % 10 == 0)
+                        Thread.sleep(20)
+
+                })
+        listener.publish(new AllFilesLoadedEvent())
+    }
+
+    private void persistFile(SharedFile sf) {
+        persisterExecutor.submit({
+            def jsonPath = getJsonPath(sf)
+
+            def startTime = System.currentTimeMillis()
+            jsonPath.parent.toFile().mkdirs()
+            jsonPath.toFile().withPrintWriter { writer ->
+                def json = toJson sf
+                json = JsonOutput.toJson(json)
+                writer.println json
+            }
+            log.fine("Time(ms) to write json: " + (System.currentTimeMillis() - startTime))
+        } as Runnable)
+    }
+    private Path getJsonPath(SharedFile sf){
+        def pathHash = sf.getB64PathHash()
+        return Paths.get(
+                location.getAbsolutePath(),
+                pathHash.substring(0, CUT_LENGTH),
+                pathHash.substring(CUT_LENGTH) + ".json"
+        )
+    }
+}

core/src/main/groovy/com/muwire/core/files/PersisterService.groovy

@@ -1,39 +1,24 @@
 package com.muwire.core.files
 
-import java.nio.file.CopyOption
-import java.nio.file.Files
-import java.nio.file.StandardCopyOption
+
 import java.util.concurrent.ExecutorService
 import java.util.concurrent.Executors
 import java.util.concurrent.ThreadFactory
 import java.util.logging.Level
-import java.util.stream.Collectors
 
-import com.muwire.core.DownloadedFile
 import com.muwire.core.EventBus
-import com.muwire.core.InfoHash
-import com.muwire.core.Service
-import com.muwire.core.SharedFile
 import com.muwire.core.UILoadedEvent
-import com.muwire.core.util.DataUtil
-
-import groovy.json.JsonOutput
 import groovy.json.JsonSlurper
 import groovy.util.logging.Log
-import net.i2p.data.Base64
-import net.i2p.data.Destination
 
 @Log
-class PersisterService extends Service {
+class PersisterService extends BasePersisterService {
 
     final File location
     final EventBus listener
     final int interval
     final Timer timer
     final FileManager fileManager
-    final ExecutorService persisterExecutor = Executors.newSingleThreadExecutor({ r -> 
-        new Thread(r, "file persister")
-    } as ThreadFactory)
 
     PersisterService(File location, EventBus listener, int interval, FileManager fileManager) {
         this.location = location
@@ -50,10 +35,6 @@ class PersisterService extends Service {
     void onUILoadedEvent(UILoadedEvent e) {
         timer.schedule({load()} as TimerTask, 1)
     }
-    
-    void onUIPersistFilesEvent(UIPersistFilesEvent e) {
-        persistFiles()
-    }
 
     void load() {
         Thread.currentThread().setPriority(Thread.MIN_PRIORITY)
@@ -68,6 +49,7 @@ class PersisterService extends Service {
                         def event = fromJson parsed
                         if (event != null) {
                             log.fine("loaded file $event.loadedFile.file")
+                            event.source = "PersisterService"
                             listener.publish event
                             loaded++
                             if (loaded % 10 == 0)
@@ -75,107 +57,18 @@ class PersisterService extends Service {
                         }
                     }
                 }
-                listener.publish(new AllFilesLoadedEvent())
-            } catch (IllegalArgumentException|NumberFormatException e) {
+                // Backup the old hashes
+                location.renameTo(
+                        new File(location.absolutePath + ".bak")
+                )
+                listener.publish(new PersisterDoneEvent())
+            } catch (IllegalArgumentException e) {
                 log.log(Level.WARNING, "couldn't load files",e)
             }
         } else {
-            listener.publish(new AllFilesLoadedEvent())
+            listener.publish(new PersisterDoneEvent())
         }
-        timer.schedule({persistFiles()} as TimerTask, 1000, interval)
         loaded = true
     }
 
-    private static FileLoadedEvent fromJson(def json) {
-        if (json.file == null || json.length == null || json.infoHash == null || json.hashList == null)
-            throw new IllegalArgumentException()
-        if (!(json.hashList instanceof List))
-            throw new IllegalArgumentException()
-
-        def file = new File(DataUtil.readi18nString(Base64.decode(json.file)))
-        file = file.getCanonicalFile()
-        if (!file.exists() || file.isDirectory())
-            return null
-        long length = Long.valueOf(json.length)
-        if (length != file.length())
-            return null
-
-        List hashList = (List) json.hashList
-        ByteArrayOutputStream baos = new ByteArrayOutputStream()
-        hashList.each {
-            byte [] hash = Base64.decode it.toString()
-            if (hash == null)
-                throw new IllegalArgumentException()
-            baos.write hash
-        }
-        byte[] hashListBytes = baos.toByteArray()
-
-        InfoHash ih = InfoHash.fromHashList(hashListBytes)
-        byte [] root = Base64.decode(json.infoHash.toString())
-        if (root == null)
-            throw new IllegalArgumentException()
-        if (!Arrays.equals(root, ih.getRoot()))
-            return null
-
-        int pieceSize = 0
-        if (json.pieceSize != null)
-            pieceSize = json.pieceSize
-
-        if (json.sources != null) {
-            List sources = (List)json.sources
-            Set<Destination> sourceSet = sources.stream().map({d -> new Destination(d.toString())}).collect Collectors.toSet()
-            DownloadedFile df = new DownloadedFile(file, ih, pieceSize, sourceSet)
-            df.setComment(json.comment)
-            return new FileLoadedEvent(loadedFile : df)
-        }
-
-        int hits = 0
-        if (json.hits != null)
-            hits = json.hits
-
-        SharedFile sf = new SharedFile(file, ih, pieceSize)
-        sf.setComment(json.comment)
-        sf.hits = hits
-        if (json.downloaders != null)
-            sf.getDownloaders().addAll(json.downloaders)
-        return new FileLoadedEvent(loadedFile: sf)
-
-    }
-
-    private void persistFiles() {
-        persisterExecutor.submit( {
-            def sharedFiles = fileManager.getSharedFiles()
-
-            File tmp = File.createTempFile("muwire-files", "tmp")
-            tmp.deleteOnExit()
-            tmp.withPrintWriter { writer ->
-                sharedFiles.each { k, v ->
-                    def json = toJson(k,v)
-                    json = JsonOutput.toJson(json)
-                    writer.println json
-                }
-            }
-            Files.copy(tmp.toPath(), location.toPath(), StandardCopyOption.REPLACE_EXISTING)
-            tmp.delete()
-        } as Runnable)
-    }
-
-    private def toJson(File f, SharedFile sf) {
-        def json = [:]
-        json.file = sf.getB64EncodedFileName()
-        json.length = sf.getCachedLength()
-        InfoHash ih = sf.getInfoHash()
-        json.infoHash = sf.getB64EncodedHashRoot()
-        json.pieceSize = sf.getPieceSize()
-        json.hashList = sf.getB64EncodedHashList()
-        json.comment = sf.getComment()
-        json.hits = sf.getHits()
-        json.downloaders = sf.getDownloaders()
-
-        if (sf instanceof DownloadedFile) {
-            json.sources = sf.sources.stream().map( {d -> d.toBase64()}).collect(Collectors.toList())
-        }
-
-        json
-    }
 }

core/src/main/groovy/com/muwire/core/files/UIPersistFilesEvent.groovy

@@ -1,6 +0,0 @@
-package com.muwire.core.files
-
-import com.muwire.core.Event
-
-class UIPersistFilesEvent extends Event {
-}

core/src/main/groovy/com/muwire/core/hostcache/CacheClient.groovy

@@ -1,5 +1,7 @@
 package com.muwire.core.hostcache
 
+import java.util.concurrent.atomic.AtomicBoolean
+
 import com.muwire.core.EventBus
 import com.muwire.core.MuWireSettings
 import com.muwire.core.connection.ConnectionManager
@@ -27,6 +29,7 @@ class CacheClient {
     final long interval
     final MuWireSettings settings
     final Timer timer
+    private final AtomicBoolean stopped = new AtomicBoolean();
 
     public CacheClient(EventBus eventBus, HostCache cache,
         ConnectionManager manager, I2PSession session,
@@ -47,9 +50,12 @@ class CacheClient {
 
     void stop() {
         timer.cancel()
+        stopped.set(true)
     }
 
     private void queryIfNeeded() {
+        if (stopped.get())
+            return
         if (!manager.getConnections().isEmpty())
             return
         if (!cache.getHosts(1).isEmpty())
@@ -65,7 +71,12 @@ class CacheClient {
         options.setSendLeaseSet(true)
         CacheServers.getCacheServers().each {
             log.info "Querying hostcache ${it.toBase32()}"
-            session.sendMessage(it, ping, 0, ping.length, I2PSession.PROTO_DATAGRAM, 1, 0, options)
+            try {
+                session.sendMessage(it, ping, 0, ping.length, I2PSession.PROTO_DATAGRAM, 1, 0, options)
+            } catch (Exception e) {
+                if (!stopped.get())
+                    throw e
+            }
         }
     }
 

core/src/main/groovy/com/muwire/core/search/BrowseManager.groovy

@@ -2,6 +2,7 @@ package com.muwire.core.search
 
 import com.muwire.core.Constants
 import com.muwire.core.EventBus
+import com.muwire.core.Persona
 import com.muwire.core.connection.Endpoint
 import com.muwire.core.connection.I2PConnector
 import com.muwire.core.util.DataUtil
@@ -20,22 +21,26 @@ class BrowseManager {
     
     private final I2PConnector connector
     private final EventBus eventBus
+    private final Persona me
     
     private final Executor browserThread = Executors.newSingleThreadExecutor()
     
-    BrowseManager(I2PConnector connector, EventBus eventBus) {
+    BrowseManager(I2PConnector connector, EventBus eventBus, Persona me) {
         this.connector = connector
         this.eventBus = eventBus
+        this.me = me
     }
     
     void onUIBrowseEvent(UIBrowseEvent e) {
         browserThread.execute({
             Endpoint endpoint = null
             try {
-                eventBus.publish(new BrowseStatusEvent(status : BrowseStatus.CONNECTING))
+                eventBus.publish(new BrowseStatusEvent(host : e.host, status : BrowseStatus.CONNECTING))
                 endpoint = connector.connect(e.host.destination)
                 OutputStream os = endpoint.getOutputStream()
-                os.write("BROWSE\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+                os.write("BROWSE\r\n".getBytes(StandardCharsets.US_ASCII))
+                os.write("Persona:${me.toBase64()}\r\n".getBytes(StandardCharsets.US_ASCII))
+                os.write("\r\n".getBytes(StandardCharsets.US_ASCII))
              
                 InputStream is = endpoint.getInputStream()
                 String code = DataUtil.readTillRN(is)
@@ -43,24 +48,17 @@ class BrowseManager {
                     throw new IOException("Invalid code $code")
                     
                 // parse all headers
-                Map<String,String> headers = new HashMap<>()
-                String header
-                while((header = DataUtil.readTillRN(is)) != "" && headers.size() < Constants.MAX_HEADERS) {
-                    int colon = header.indexOf(':')
-                    if (colon == -1 || colon == header.length() - 1)
-                        throw new IOException("invalid header $header")
-                    String key = header.substring(0, colon)
-                    String value = header.substring(colon + 1)
-                    headers[key] = value.trim()
-                }
+                Map<String,String> headers = DataUtil.readAllHeaders(is)
                 
                 if (!headers.containsKey("Count"))
                     throw new IOException("No count header")
                 
                 int results = Integer.parseInt(headers['Count'])
                 
+                boolean chat = headers.containsKey("Chat") && Boolean.parseBoolean(headers['Chat'])
+                
                 // at this stage, start pulling the results
-                eventBus.publish(new BrowseStatusEvent(status : BrowseStatus.FETCHING, totalResults : results))
+                eventBus.publish(new BrowseStatusEvent(host: e.host, status : BrowseStatus.FETCHING, totalResults : results))
                 
                 JsonSlurper slurper = new JsonSlurper()
                 DataInputStream dis = new DataInputStream(new GZIPInputStream(is))
@@ -71,14 +69,15 @@ class BrowseManager {
                     dis.readFully(tmp)
                     def json = slurper.parse(tmp)
                     UIResultEvent result = ResultsParser.parse(e.host, uuid, json)
+                    result.chat = chat
                     eventBus.publish(result)
                 }
                 
-                eventBus.publish(new BrowseStatusEvent(status : BrowseStatus.FINISHED))
+                eventBus.publish(new BrowseStatusEvent(host: e.host, status : BrowseStatus.FINISHED))
                 
             } catch (Exception bad) {
                 log.log(Level.WARNING, "browse failed", bad)
-                eventBus.publish(new BrowseStatusEvent(status : BrowseStatus.FAILED))
+                eventBus.publish(new BrowseStatusEvent(host: e.host, status : BrowseStatus.FAILED))
             } finally {
                 endpoint?.close()
             }

core/src/main/groovy/com/muwire/core/search/BrowseStatusEvent.groovy

@@ -1,8 +1,10 @@
 package com.muwire.core.search
 
 import com.muwire.core.Event
+import com.muwire.core.Persona
 
 class BrowseStatusEvent extends Event {
+    Persona host
     BrowseStatus status
     int totalResults
 }

core/src/main/groovy/com/muwire/core/search/QueryEvent.groovy

@@ -13,6 +13,8 @@ class QueryEvent extends Event {
     Persona originator
     Destination receivedOn
     byte[] sig
+    long queryTime
+    byte[] sig2
 
     String toString() {
         "searchEvent: $searchEvent firstHop:$firstHop, replyTo:${replyTo.toBase32()}" +

core/src/main/groovy/com/muwire/core/search/ResultsSender.groovy

@@ -1,6 +1,7 @@
 package com.muwire.core.search
 
 import com.muwire.core.SharedFile
+import com.muwire.core.chat.ChatServer
 import com.muwire.core.connection.Endpoint
 import com.muwire.core.connection.I2PConnector
 import com.muwire.core.filecert.CertificateManager
@@ -48,13 +49,16 @@ class ResultsSender {
     private final EventBus eventBus
     private final MuWireSettings settings
     private final CertificateManager certificateManager
+    private final ChatServer chatServer
 
-    ResultsSender(EventBus eventBus, I2PConnector connector, Persona me, MuWireSettings settings, CertificateManager certificateManager) {
+    ResultsSender(EventBus eventBus, I2PConnector connector, Persona me, MuWireSettings settings, 
+        CertificateManager certificateManager, ChatServer chatServer) {
         this.connector = connector;
         this.eventBus = eventBus
         this.me = me
         this.settings = settings
         this.certificateManager = certificateManager
+        this.chatServer = chatServer
     }
 
     void sendResults(UUID uuid, SharedFile[] results, Destination target, boolean oobInfohash, boolean compressedResults) {
@@ -80,9 +84,11 @@ class ResultsSender {
                     infohash : it.getInfoHash(),
                     pieceSize : pieceSize,
                     uuid : uuid,
+                    browse : settings.browseFiles,
                     sources : suggested,
                     comment : comment,
-                    certificates : certificates
+                    certificates : certificates,
+                    chat : chatServer.running.get() && settings.advertiseChat
                     )
                 uiResultEvents << uiResultEvent
             }
@@ -130,6 +136,8 @@ class ResultsSender {
                         os.write("RESULTS $uuid\r\n".getBytes(StandardCharsets.US_ASCII))
                         os.write("Sender: ${me.toBase64()}\r\n".getBytes(StandardCharsets.US_ASCII))
                         os.write("Count: $results.length\r\n".getBytes(StandardCharsets.US_ASCII))
+                        boolean chat = chatServer.running.get() && settings.advertiseChat
+                        os.write("Chat: $chat\r\n".getBytes(StandardCharsets.US_ASCII))
                         os.write("\r\n".getBytes(StandardCharsets.US_ASCII))
                         DataOutputStream dos = new DataOutputStream(new GZIPOutputStream(os))
                         results.each { 

core/src/main/groovy/com/muwire/core/search/SearchEvent.groovy

@@ -2,6 +2,7 @@ package com.muwire.core.search
 
 import com.muwire.core.Event
 import com.muwire.core.InfoHash
+import com.muwire.core.Persona
 
 class SearchEvent extends Event {
 
@@ -11,6 +12,7 @@ class SearchEvent extends Event {
     boolean oobInfohash
     boolean searchComments
     boolean compressedResults
+    Persona persona
 
     String toString() {
         def infoHash = null

core/src/main/groovy/com/muwire/core/search/SearchIndex.groovy

@@ -39,10 +39,11 @@ class SearchIndex {
         split.each { if (it.length() > 0) rv << it }
         
         // then just by ' '
-        source.split(' ').each { if (it.length() > 0) rv << it }
+        source.toLowerCase().split(' ').each { if (it.length() > 0) rv << it }
         
         // and add original string
         rv << source
+        rv << source.toLowerCase()
         rv.toArray(new String[0])
     }
 

core/src/main/groovy/com/muwire/core/search/UIResultEvent.groovy

@@ -17,6 +17,7 @@ class UIResultEvent extends Event {
     String comment
     boolean browse
     int certificates
+    boolean chat
 
     @Override
     public String toString() {

core/src/main/groovy/com/muwire/core/trust/RemoteTrustList.groovy

@@ -3,14 +3,15 @@ package com.muwire.core.trust
 import java.util.concurrent.ConcurrentHashMap
 
 import com.muwire.core.Persona
+import com.muwire.core.trust.TrustService.TrustEntry
 
 import net.i2p.util.ConcurrentHashSet
 
 class RemoteTrustList {
     public enum Status { NEW, UPDATING, UPDATED, UPDATE_FAILED }
 
-    private final Persona persona
-    private final Set<Persona> good, bad
+    final Persona persona
+    final Set<TrustEntry> good, bad
     volatile long timestamp
     volatile boolean forceUpdate
     Status status = Status.NEW

core/src/main/groovy/com/muwire/core/trust/TrustEvent.groovy

@@ -7,4 +7,5 @@ class TrustEvent extends Event {
 
     Persona persona
     TrustLevel level
+    String reason
 }

core/src/main/groovy/com/muwire/core/trust/TrustService.groovy

@@ -1,21 +1,26 @@
 package com.muwire.core.trust
 
 import java.util.concurrent.ConcurrentHashMap
+import java.util.logging.Level
 
 import com.muwire.core.Persona
 import com.muwire.core.Service
 
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
+import groovy.util.logging.Log
 import net.i2p.data.Base64
 import net.i2p.data.Destination
 import net.i2p.util.ConcurrentHashSet
 
+@Log
 class TrustService extends Service {
 
     final File persistGood, persistBad
     final long persistInterval
 
-    final Map<Destination, Persona> good = new ConcurrentHashMap<>()
-    final Map<Destination, Persona> bad = new ConcurrentHashMap<>()
+    final Map<Destination, TrustEntry> good = new ConcurrentHashMap<>()
+    final Map<Destination, TrustEntry> bad = new ConcurrentHashMap<>()
 
     final Timer timer
 
@@ -37,18 +42,41 @@ class TrustService extends Service {
     }
 
     void load() {
+        JsonSlurper slurper = new JsonSlurper()
         if (persistGood.exists()) {
             persistGood.eachLine {
-                byte [] decoded = Base64.decode(it)
-                Persona persona = new Persona(new ByteArrayInputStream(decoded))
-                good.put(persona.destination, persona)
+                try {
+                    byte [] decoded = Base64.decode(it)
+                    Persona persona = new Persona(new ByteArrayInputStream(decoded))
+                    good.put(persona.destination, new TrustEntry(persona, null))
+                } catch (Exception e) {
+                    try {
+                        def json = slurper.parseText(it)
+                        byte [] decoded = Base64.decode(json.persona)
+                        Persona persona = new Persona(new ByteArrayInputStream(decoded))
+                        good.put(persona.destination, new TrustEntry(persona, json.reason))
+                    } catch (Exception bad) {
+                        log.log(Level.WARNING,"couldn't parse trust entry $it",bad)
+                    }
+                }
             }
         }
         if (persistBad.exists()) {
             persistBad.eachLine {
-                byte [] decoded = Base64.decode(it)
-                Persona persona = new Persona(new ByteArrayInputStream(decoded))
-                bad.put(persona.destination, persona)
+                 try {
+                    byte [] decoded = Base64.decode(it)
+                    Persona persona = new Persona(new ByteArrayInputStream(decoded))
+                    bad.put(persona.destination, new TrustEntry(persona, null))
+                } catch (Exception e) {
+                    try {
+                        def json = slurper.parseText(it)
+                        byte [] decoded = Base64.decode(json.persona)
+                        Persona persona = new Persona(new ByteArrayInputStream(decoded))
+                        bad.put(persona.destination, new TrustEntry(persona, json.reason))
+                    } catch (Exception bad) {
+                        log.log(Level.WARNING,"couldn't parse trust entry $it",bad)
+                    }
+                }
             }
         }
         timer.schedule({persist()} as TimerTask, persistInterval, persistInterval)
@@ -59,13 +87,19 @@ class TrustService extends Service {
         persistGood.delete()
         persistGood.withPrintWriter { writer ->
             good.each {k,v ->
-                writer.println v.toBase64()
+                def json = [:]
+                json.persona = v.persona.toBase64()
+                json.reason = v.reason
+                writer.println JsonOutput.toJson(json)
             }
         }
         persistBad.delete()
         persistBad.withPrintWriter { writer ->
             bad.each { k,v ->
-                writer.println v.toBase64()
+                def json = [:]
+                json.persona = v.persona.toBase64()
+                json.reason = v.reason
+                writer.println JsonOutput.toJson(json)
             }
         }
     }
@@ -82,11 +116,11 @@ class TrustService extends Service {
         switch(e.level) {
             case TrustLevel.TRUSTED:
                 bad.remove(e.persona.destination)
-                good.put(e.persona.destination, e.persona)
+                good.put(e.persona.destination, new TrustEntry(e.persona, e.reason))
                 break
             case TrustLevel.DISTRUSTED:
                 good.remove(e.persona.destination)
-                bad.put(e.persona.destination, e.persona)
+                bad.put(e.persona.destination, new TrustEntry(e.persona, e.reason))
                 break
             case TrustLevel.NEUTRAL:
                 good.remove(e.persona.destination)
@@ -94,4 +128,24 @@ class TrustService extends Service {
                 break
         }
     }
+    
+    public static class TrustEntry {
+        final Persona persona
+        final String reason
+        TrustEntry(Persona persona, String reason) {
+            this.persona = persona
+            this.reason = reason
+        }
+        
+        public int hashCode() {
+            persona.hashCode()
+        }
+        
+        public boolean equals(Object o) {
+            if (!(o instanceof TrustEntry))
+                return false
+            persona == o.persona
+        }
+    }
+    
 }

core/src/main/groovy/com/muwire/core/trust/TrustSubscriber.groovy

@@ -12,9 +12,12 @@ import com.muwire.core.Persona
 import com.muwire.core.UILoadedEvent
 import com.muwire.core.connection.Endpoint
 import com.muwire.core.connection.I2PConnector
+import com.muwire.core.trust.TrustService.TrustEntry
 import com.muwire.core.util.DataUtil
 
+import groovy.json.JsonSlurper
 import groovy.util.logging.Log
+import net.i2p.data.Base64
 import net.i2p.data.Destination
 
 @Log
@@ -23,7 +26,7 @@ class TrustSubscriber {
     private final I2PConnector i2pConnector
     private final MuWireSettings settings
 
-    private final Map<Destination, RemoteTrustList> remoteTrustLists = new ConcurrentHashMap<>()
+    final Map<Destination, RemoteTrustList> remoteTrustLists = new ConcurrentHashMap<>()
 
     private final Object waitLock = new Object()
     private volatile boolean shutdown
@@ -47,7 +50,7 @@ class TrustSubscriber {
         thread?.interrupt()
         updateThreads.shutdownNow()
     }
-
+    
     void onTrustSubscriptionEvent(TrustSubscriptionEvent e) {
         if (!e.subscribe) {
             remoteTrustLists.remove(e.persona.destination)
@@ -59,6 +62,10 @@ class TrustSubscriber {
             }
         }
     }
+    
+    public boolean isSubscribed(Persona p) {
+        remoteTrustLists.containsKey(p.destination)
+    }
 
     private void checkLoop() {
         try {
@@ -109,7 +116,9 @@ class TrustSubscriber {
             endpoint = i2pConnector.connect(trustList.persona.destination)
             OutputStream os = endpoint.getOutputStream()
             InputStream is = endpoint.getInputStream()
-            os.write("TRUST\r\n\r\n".getBytes(StandardCharsets.US_ASCII))
+            os.write("TRUST\r\n".getBytes(StandardCharsets.US_ASCII))
+            os.write("Json:true\r\n".getBytes(StandardCharsets.US_ASCII))
+            os.write("\r\n".getBytes(StandardCharsets.US_ASCII))
             os.flush()
 
             String codeString = DataUtil.readTillRN(is)
@@ -123,24 +132,47 @@ class TrustSubscriber {
                 return false
             }
 
-            // swallow any headers
-            String header
-            while (( header = DataUtil.readTillRN(is)) != "");
-
+            Map<String,String> headers = DataUtil.readAllHeaders(is)
             DataInputStream dis = new DataInputStream(is)
+            Set<TrustService.TrustEntry> good = new HashSet<>()
+            Set<TrustService.TrustEntry> bad = new HashSet<>()
+            
+            if (headers.containsKey('Json') && Boolean.parseBoolean(headers['Json'])) {
+                int countGood = Integer.parseInt(headers['Good'])
+                int countBad = Integer.parseInt(headers['Bad'])
+                
+                JsonSlurper slurper = new JsonSlurper()
+                
+                for (int i = 0; i < countGood; i++) {
+                    int length = dis.readUnsignedShort()
+                    byte []payload = new byte[length]
+                    dis.readFully(payload)
+                    def json = slurper.parse(payload)
+                    Persona persona = new Persona(new ByteArrayInputStream(Base64.decode(json.persona)))
+                    good.add(new TrustEntry(persona, json.reason))
+                }
+                
+                for (int i = 0; i < countBad; i++) {
+                    int length = dis.readUnsignedShort()
+                    byte []payload = new byte[length]
+                    dis.readFully(payload)
+                    def json = slurper.parse(payload)
+                    Persona persona = new Persona(new ByteArrayInputStream(Base64.decode(json.persona)))
+                    bad.add(new TrustEntry(persona, json.reason))
+                }
+                
+            } else {
+                int nGood = dis.readUnsignedShort()
+                for (int i = 0; i < nGood; i++) {
+                    Persona p = new Persona(dis)
+                    good.add(new TrustEntry(p,null))
+                }
 
-            Set<Persona> good = new HashSet<>()
-            int nGood = dis.readUnsignedShort()
-            for (int i = 0; i < nGood; i++) {
-                Persona p = new Persona(dis)
-                good.add(p)
-            }
-
-            Set<Persona> bad = new HashSet<>()
-            int nBad = dis.readUnsignedShort()
-            for (int i = 0; i < nBad; i++) {
-                Persona p = new Persona(dis)
-                bad.add(p)
+                int nBad = dis.readUnsignedShort()
+                for (int i = 0; i < nBad; i++) {
+                    Persona p = new Persona(dis)
+                    bad.add(new TrustEntry(p, null))
+                }
             }
 
             trustList.timestamp = now

core/src/main/groovy/com/muwire/core/update/UpdateClient.groovy

@@ -13,6 +13,7 @@ import com.muwire.core.files.FileSharedEvent
 import com.muwire.core.search.QueryEvent
 import com.muwire.core.search.SearchEvent
 import com.muwire.core.search.UIResultBatchEvent
+import com.muwire.core.util.DataUtil
 
 import groovy.json.JsonOutput
 import groovy.json.JsonSlurper
@@ -176,9 +177,12 @@ class UpdateClient {
                         signer = payload.signer
                         log.info("starting search for new version hash $payload.infoHash")
                         Signature sig = DSAEngine.getInstance().sign(updateInfoHash.getRoot(), spk)
-                        def searchEvent = new SearchEvent(searchHash : updateInfoHash.getRoot(), uuid : UUID.randomUUID(), oobInfohash : true)
+                        UUID uuid = UUID.randomUUID()
+                        long timestamp = System.currentTimeMillis()
+                        byte [] sig2 = DataUtil.signUUID(uuid, timestamp, spk)
+                        def searchEvent = new SearchEvent(searchHash : updateInfoHash.getRoot(), uuid : uuid, oobInfohash : true, persona : me)
                         def queryEvent = new QueryEvent(searchEvent : searchEvent, firstHop : true, replyTo : me.destination,
-                            receivedOn : me.destination, originator : me, sig : sig.data)
+                            receivedOn : me.destination, originator : me, sig : sig.data, queryTime : timestamp, sig2 : sig2)
                         eventBus.publish(queryEvent)
                     }
                 }

core/src/main/java/com/muwire/core/Constants.java

@@ -4,13 +4,18 @@ import net.i2p.crypto.SigType;
 
 public class Constants {
     public static final byte PERSONA_VERSION = (byte)1;
-    public static final byte FILE_CERT_VERSION = (byte)1;
+    public static final byte FILE_CERT_VERSION = (byte)2;
+    public static final int CHAT_VERSION = 1;
+    
     public static final SigType SIG_TYPE = SigType.EdDSA_SHA512_Ed25519;
 
     public static final int MAX_HEADER_SIZE = 0x1 << 14;
     public static final int MAX_HEADERS = 16;
+    public static final long MAX_HEADER_TIME = 60 * 1000;
     
     public static final int MAX_RESULTS = 0x1 << 16;
     
     public static final int MAX_COMMENT_LENGTH = 0x1 << 15;
+    
+    public static final long MAX_QUERY_AGE = 5 * 60 * 1000L; 
 }

core/src/main/java/com/muwire/core/InvalidSignatureException.java

@@ -1,4 +1,4 @@
-package com.muwire.core
+package com.muwire.core;
 
 class InvalidSignatureException extends Exception {
 

core/src/main/java/com/muwire/core/Name.java

@@ -0,0 +1,51 @@
+package com.muwire.core;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * A name of persona, file or search term
+ */
+public class Name {
+    final String name;
+
+    Name(String name) {
+        this.name = name;
+    }
+
+    Name(InputStream nameStream) throws IOException {
+        DataInputStream dis = new DataInputStream(nameStream);
+        int length = dis.readUnsignedShort();
+        byte [] nameBytes = new byte[length];
+        dis.readFully(nameBytes);
+        this.name = new String(nameBytes, StandardCharsets.UTF_8);
+    }
+
+    public void write(OutputStream out) throws IOException {
+        DataOutputStream dos = new DataOutputStream(out);
+        byte [] bytes = name.getBytes(StandardCharsets.UTF_8);
+        dos.writeShort(bytes.length);
+        dos.write(bytes);
+    }
+
+    public String getName() {
+        return name;
+    }
+
+    @Override
+    public int hashCode() {
+        return name.hashCode();
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (!(o instanceof Name))
+            return false;
+        Name other = (Name)o;
+        return name.equals(other.name);
+    }
+}

core/src/main/java/com/muwire/core/Persona.java

@@ -0,0 +1,110 @@
+package com.muwire.core;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import net.i2p.crypto.DSAEngine;
+import net.i2p.data.Base64;
+import net.i2p.data.DataFormatException;
+import net.i2p.data.Destination;
+import net.i2p.data.Signature;
+import net.i2p.data.SigningPublicKey;
+
+public class Persona {
+    private static final int SIG_LEN = Constants.SIG_TYPE.getSigLen();
+
+    private final byte version;
+    private final Name name;
+    private final Destination destination;
+    private final byte[] sig;
+    private volatile String humanReadableName;
+    private volatile String base64;
+    private volatile byte[] payload;
+
+    public Persona(InputStream personaStream) throws IOException, DataFormatException, InvalidSignatureException {
+        version = (byte) (personaStream.read() & 0xFF);
+        if (version != Constants.PERSONA_VERSION)
+            throw new IOException("Unknown version "+version);
+
+        name = new Name(personaStream);
+        destination = Destination.create(personaStream);
+        sig = new byte[SIG_LEN];
+        DataInputStream dis = new DataInputStream(personaStream);
+        dis.readFully(sig);
+        if (!verify(version, name, destination, sig))
+            throw new InvalidSignatureException(getHumanReadableName() + " didn't verify");
+    }
+
+    private static boolean verify(byte version, Name name, Destination destination, byte [] sig) 
+    throws IOException, DataFormatException {
+        ByteArrayOutputStream baos = new ByteArrayOutputStream();
+        baos.write(version);
+        name.write(baos);
+        destination.writeBytes(baos);
+        byte[] payload = baos.toByteArray();
+        SigningPublicKey spk = destination.getSigningPublicKey();
+        Signature signature = new Signature(Constants.SIG_TYPE, sig);
+        return DSAEngine.getInstance().verifySignature(signature, payload, spk);
+    }
+
+    public void write(OutputStream out) throws IOException, DataFormatException {
+        if (payload == null) {
+            ByteArrayOutputStream baos = new ByteArrayOutputStream();
+            baos.write(version);
+            name.write(baos);
+            destination.writeBytes(baos);
+            baos.write(sig);
+            payload = baos.toByteArray();
+        }
+        out.write(payload);
+    }
+
+    public String getHumanReadableName() {
+        if (humanReadableName == null)
+            humanReadableName = name.getName() + "@" + destination.toBase32().substring(0,32);
+        return humanReadableName;
+    }
+    
+    public Destination getDestination() {
+        return destination;
+    }
+
+    public String toBase64() {
+        if (base64 == null) {
+            ByteArrayOutputStream baos = new ByteArrayOutputStream();
+            try {
+                write(baos);
+            } catch (Exception impossible) {
+                throw new RuntimeException(impossible);
+            }
+            base64 = Base64.encode(baos.toByteArray());
+        }
+        return base64;
+    }
+
+    @Override
+    public int hashCode() {
+        return name.hashCode() ^ destination.hashCode();
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (!(o instanceof Persona))
+            return false;
+        Persona other = (Persona)o;
+        return name.equals(other.name) && destination.equals(other.destination);
+    }
+
+    public static void main(String []args) throws Exception {
+        if (args.length != 1) {
+            System.out.println("This utility decodes a bas64-encoded persona");
+            System.exit(1);
+        }
+        Persona p = new Persona(new ByteArrayInputStream(Base64.decode(args[0])));
+        System.out.println(p.getHumanReadableName());
+    }
+}

core/src/main/java/com/muwire/core/SharedFile.java

@@ -2,10 +2,13 @@ package com.muwire.core;
 
 import java.io.File;
 import java.io.IOException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Objects;
 import java.util.Set;
 
 import com.muwire.core.util.DataUtil;
@@ -20,14 +23,15 @@ public class SharedFile {
 
     private final String cachedPath;
     private final long cachedLength;
-    
+
+    private String b64PathHash;
     private final String b64EncodedFileName;
     private final String b64EncodedHashRoot;
     private final List<String> b64EncodedHashList;
     
     private volatile String comment;
-    private volatile int hits;
     private final Set<String> downloaders = Collections.synchronizedSet(new HashSet<>());
+    private final Set<SearchEntry> searches = Collections.synchronizedSet(new HashSet<>());
 
     public SharedFile(File file, InfoHash infoHash, int pieceSize) throws IOException {
         this.file = file;
@@ -37,7 +41,7 @@ public class SharedFile {
         this.cachedLength = file.length();
         this.b64EncodedFileName = Base64.encode(DataUtil.encodei18nString(file.toString()));
         this.b64EncodedHashRoot = Base64.encode(infoHash.getRoot());
-        
+
         List<String> b64List = new ArrayList<String>();
         byte[] tmp = new byte[32];
         for (int i = 0; i < infoHash.getHashList().length / 32; i++) {
@@ -51,6 +55,19 @@ public class SharedFile {
         return file;
     }
 
+    public byte[] getPathHash() throws NoSuchAlgorithmException {
+        var digester = MessageDigest.getInstance("SHA-256");
+        digester.update(file.getAbsolutePath().getBytes());
+        return digester.digest();
+    }
+
+    public String getB64PathHash() throws NoSuchAlgorithmException {
+        if(b64PathHash == null){
+            b64PathHash = Base64.encode(getPathHash());
+        }
+        return b64PathHash;
+    }
+
     public InfoHash getInfoHash() {
         return infoHash;
     }
@@ -97,17 +114,21 @@ public class SharedFile {
     }
     
     public int getHits() {
-        return hits;
+        return searches.size();
     }
     
-    public void hit() {
-        hits++;
+    public void hit(Persona searcher, long timestamp, String query) {
+        searches.add(new SearchEntry(searcher, timestamp, query));
     }
     
     public Set<String> getDownloaders() {
         return downloaders;
     }
     
+    public Set<SearchEntry> getSearches() {
+        return searches;
+    }
+    
     public void addDownloader(String name) {
         downloaders.add(name);
     }
@@ -124,4 +145,29 @@ public class SharedFile {
         SharedFile other = (SharedFile)o;
         return file.equals(other.file) && infoHash.equals(other.infoHash);
     }
+    
+    public static class SearchEntry {
+        private final Persona searcher;
+        private final long timestamp;
+        private final String query;
+        
+        public SearchEntry(Persona searcher, long timestamp, String query) {
+            this.searcher = searcher;
+            this.timestamp = timestamp;
+            this.query = query;
+        }
+        
+        public int hashCode() {
+            return Objects.hash(searcher) ^ Objects.hash(timestamp) ^ query.hashCode();
+        }
+        
+        public boolean equals(Object o) {
+            if (!(o instanceof SearchEntry))
+                return false;
+            SearchEntry other = (SearchEntry)o;
+            return Objects.equals(searcher, other.searcher) &&
+                    timestamp == other.timestamp &&
+                    query.equals(other.query);
+        }
+    }
 }

core/src/main/java/com/muwire/core/util/DataUtil.java

@@ -10,14 +10,20 @@ import java.lang.reflect.Method;
 import java.nio.ByteBuffer;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 import java.util.Properties;
 import java.util.Set;
+import java.util.UUID;
 import java.util.stream.Collectors;
 
 import com.muwire.core.Constants;
 
+import net.i2p.crypto.DSAEngine;
 import net.i2p.data.Base64;
+import net.i2p.data.Signature;
+import net.i2p.data.SigningPrivateKey;
 import net.i2p.util.ConcurrentHashSet;
 
 public class DataUtil {
@@ -57,7 +63,7 @@ public class DataUtil {
                 ((int)header[2] & 0xFF);
     }
 
-    static String readi18nString(byte [] encoded) {
+    public static String readi18nString(byte [] encoded) {
         if (encoded.length < 2)
             throw new IllegalArgumentException("encoding too short $encoded.length");
         int length = ((encoded[0] & 0xFF) << 8) | (encoded[1] & 0xFF);
@@ -85,9 +91,12 @@ public class DataUtil {
     }
 
     public static String readTillRN(InputStream is) throws IOException {
+        final long start = System.currentTimeMillis();
         ByteArrayOutputStream baos = new ByteArrayOutputStream();
         while(baos.size() < (Constants.MAX_HEADER_SIZE)) {
             int read = is.read();
+            if (System.currentTimeMillis() - start > Constants.MAX_HEADER_TIME)
+                throw new IOException("header taking too long");
             if (read == -1)
                 throw new IOException();
             if (read == '\r') {
@@ -99,6 +108,20 @@ public class DataUtil {
         }
         return new String(baos.toByteArray(), StandardCharsets.US_ASCII);
     }
+    
+    public static Map<String, String> readAllHeaders(InputStream is) throws IOException {
+        Map<String, String> headers = new HashMap<>();
+        String header;
+        while(!(header = readTillRN(is)).equals("") && headers.size() < Constants.MAX_HEADERS) {
+            int colon = header.indexOf(':');
+            if (colon == -1 || colon == header.length() - 1)
+                throw new IOException("Invalid header "+ header);
+            String key = header.substring(0, colon);
+            String value = header.substring(colon + 1);
+            headers.put(key, value.trim());
+        }
+        return headers;
+    }
 
     public static String encodeXHave(List<Integer> pieces, int totalPieces) {
         int bytes = totalPieces / 8;
@@ -187,4 +210,10 @@ public class DataUtil {
                 .collect(Collectors.joining(","));
         props.setProperty(property, encoded);
     }
+    
+    public static byte[] signUUID(UUID uuid, long timestamp, SigningPrivateKey spk) {
+        byte [] payload = (uuid.toString() + String.valueOf(timestamp)).getBytes(StandardCharsets.US_ASCII);
+        Signature sig = DSAEngine.getInstance().sign(payload, spk);
+        return sig.getData();
+    }
 }

core/src/main/resources/defaults/i2p.properties

@@ -0,0 +1,8 @@
+inbound.nickname=MuWire
+outbound.nickname=MuWire
+inbound.length=3
+inbound.quantity=4
+outbound.length=3
+outbound.quantity=4
+i2cp.tcp.host=127.0.0.1
+i2cp.tcp.port=7654

core/src/test/groovy/com/muwire/core/SplitPatternTest.groovy

@@ -24,4 +24,12 @@ class SplitPatternTest {
         assert SplitPattern.termify('"siamese cat" any cat "persian cat"') ==
             ['siamese cat','any','cat','persian cat']
     }
+    
+    @Test
+    void testNewLine() {
+        def s = "first\nsecond"
+        s = s.replaceAll(SplitPattern.SPLIT_PATTERN, " ")
+        s = s.split(" ")
+        assert s.length == 2
+    }
 }

core/src/test/groovy/com/muwire/core/connection/ConnectionAcceptorTest.groovy

@@ -95,7 +95,7 @@ class ConnectionAcceptorTest {
         connectionEstablisher = connectionEstablisherMock.proxyInstance()
 
         acceptor = new ConnectionAcceptor(eventBus, connectionManager, settings, i2pAcceptor,
-            hostCache, trustService, searchManager, uploadManager, null, connectionEstablisher, null)
+            hostCache, trustService, searchManager, uploadManager, null, connectionEstablisher, null, null)
         acceptor.start()
         Thread.sleep(100)
     }

core/src/test/groovy/com/muwire/core/files/FileManagerTest.groovy

@@ -1,5 +1,7 @@
 package com.muwire.core.files
 
+import static org.junit.jupiter.api.Assertions.assertAll
+
 import org.junit.Before
 import org.junit.Test
 
@@ -9,6 +11,9 @@ import com.muwire.core.MuWireSettings
 import com.muwire.core.SharedFile
 import com.muwire.core.search.ResultsEvent
 import com.muwire.core.search.SearchEvent
+import com.muwire.core.util.DataUtil
+
+import net.i2p.data.Base64
 
 class FileManagerTest {
 
@@ -185,4 +190,39 @@ class FileManagerTest {
         assert results == null
 
     }
+    
+    @Test
+    void testComplicatedScenario() {
+        // this tries to reproduce an NPE when un-sharing then sharing again and searching
+        String comment = "same comment"
+        comment = Base64.encode(DataUtil.encodei18nString(comment))
+        File f1 = new File("MuWire-0.5.10.AppImage")
+        InfoHash ih1 = InfoHash.fromHashList(new byte[32])
+        SharedFile sf1 = new SharedFile(f1, ih1, 0)
+        sf1.setComment(comment)
+        
+        manager.onFileLoadedEvent(new FileLoadedEvent(loadedFile : sf1))
+        manager.onFileUnsharedEvent(new FileUnsharedEvent(unsharedFile : sf1, deleted : true))
+        
+        File f2 = new File("MuWire-0.6.0.AppImage")
+        InfoHash ih2 = InfoHash.fromHashList(new byte[64])
+        SharedFile sf2 = new SharedFile(f2, ih2, 0)
+        sf2.setComment(comment)
+        
+        manager.onFileLoadedEvent(new FileLoadedEvent(loadedFile : sf2))
+        
+        manager.onSearchEvent(new SearchEvent(searchTerms : ["muwire"]))
+        Thread.sleep(20)
+        
+        assert results != null
+        assert results.results.size() == 1
+        assert results.results.contains(sf2)
+        
+        results = null
+        manager.onSearchEvent(new SearchEvent(searchTerms : ['comment'], searchComments : true, oobInfohash : true))
+        Thread.sleep(20)
+        assert results != null
+        assert results.results.size() == 1
+        assert results.results.contains(sf2)
+    }
 }

core/src/test/groovy/com/muwire/core/files/FileTreeTest.groovy

@@ -10,8 +10,8 @@ class FileTreeTest {
         File b = new File(a, "b")
         File c = new File(b, "c")
         
-        FileTree tree = new FileTree()
-        tree.add(c)
+        FileTree<Void> tree = new FileTree<>()
+        tree.add(c,null)
         
         assert tree.root.children.size() == 1
         assert tree.fileToNode.size() == 3
@@ -28,15 +28,110 @@ class FileTreeTest {
         File c = new File(b, "c")
         File d = new File(b, "d")
         
-        FileTree tree = new FileTree()
-        tree.add(c)
+        FileTree<Void> tree = new FileTree<>()
+        tree.add(c,null)
         
         assert tree.fileToNode.size() == 3
         
-        tree.add(d)
+        tree.add(d, null)
         assert tree.fileToNode.size() == 4
         
         tree.remove(d)
         assert tree.fileToNode.size() == 3
     }
+    
+    @Test
+    public void testTraverse() {
+        Stack stack = new Stack()
+        Set<String> values = new HashSet<>()
+        StringBuilder sb = new StringBuilder()
+        def cb = new FileTreeCallback<String>() {
+
+            @Override
+            public void onDirectoryEnter(File file) {
+                stack.push(file)
+            }
+
+            @Override
+            public void onDirectoryLeave() {
+                stack.pop()
+            }
+
+            @Override
+            public void onFile(File file, String value) {
+                values.add(value)
+            }
+        }
+        
+        File a = new File("a")
+        a.createNewFile()
+        File b = new File("b")
+        b.mkdir()
+        File c = new File(b, "c")
+        c.createNewFile()
+        File d = new File(b, "d")
+        d.mkdir()
+        File e = new File(d, "e")
+        e.createNewFile()
+        FileTree<String> tree = new FileTree<>()
+        
+        tree.add(a, "a")
+        tree.add(b, "b")
+        tree.add(c, "c")
+        tree.add(d, "d")
+        tree.add(e, "e")
+        
+        tree.traverse(cb)
+
+        assert stack.isEmpty()
+        assert values.size() == 3
+        assert values.contains("a")
+        assert values.contains("c")
+        assert values.contains("e")
+    }
+    
+    @Test
+    public void testList() {
+        Set<File> directories = new HashSet<>()
+        Set<String> values = new HashSet<>()
+        def cb = new FileListCallback<String>() {
+
+            @Override
+            public void onDirectory(File file) {
+                directories.add(file)
+            }
+
+            @Override
+            public void onFile(File file, String value) {
+                values.add(value)
+            }
+        }
+        
+        File a = new File("a")
+        a.createNewFile()
+        File b = new File("b")
+        b.mkdir()
+        File c = new File(b, "c")
+        c.createNewFile()
+        
+        FileTree<String> tree = new FileTree<>()
+        
+        tree.add(a, "a")
+        tree.add(b, "b")
+        tree.add(c, "c")
+        
+        tree.list(null, cb)
+        
+        assert directories.size() == 1
+        assert directories.contains(b)
+        assert values.size() == 1
+        assert values.contains("a")
+        
+        directories.clear()
+        values.clear()
+        tree.list(b, cb)
+        assert directories.isEmpty()
+        assert values.size() == 1
+        assert values.contains("c")
+    }
 }

core/src/test/groovy/com/muwire/core/search/SearchIndexTest.groovy

@@ -120,4 +120,26 @@ class SearchIndexTest {
         assert index.search(['cat', 'video siamese']).size() == 0
         assert index.search(['cat','video','siamese']).size() == 3
     }
+    
+    @Test
+    void testNewLine() {
+        initIndex(['first\nsecond'])
+        assert index.search(['first']).size() == 1
+        assert index.search(['second']).size() == 1
+        assert index.search(['first','second']).size() == 1
+        assert index.search(['second','first']).size() == 1
+        assert index.search(['second first']).size() == 0
+        assert index.search(['first second']).size() == 0
+    }
+    
+    @Test
+    void testDosNewLine() {
+        initIndex(['first\r\nsecond'])
+        assert index.search(['first']).size() == 1
+        assert index.search(['second']).size() == 1
+        assert index.search(['first','second']).size() == 1
+        assert index.search(['second','first']).size() == 1
+        assert index.search(['second first']).size() == 0
+        assert index.search(['first second']).size() == 0
+    }
 }

core/src/test/groovy/com/muwire/core/trust/TrustServiceTest.groovy

@@ -8,6 +8,7 @@ import com.muwire.core.Destinations
 import com.muwire.core.Persona
 import com.muwire.core.Personas
 
+import groovy.json.JsonSlurper
 import net.i2p.data.Base64
 import net.i2p.data.Destination
 
@@ -55,13 +56,16 @@ class TrustServiceTest {
         service.onTrustEvent new TrustEvent(level: TrustLevel.DISTRUSTED, persona: personas.persona2)
 
         Thread.sleep(250)
+        JsonSlurper slurper = new JsonSlurper()
         def trusted = new HashSet<>()
         persistGood.eachLine {
-            trusted.add(new Persona(new ByteArrayInputStream(Base64.decode(it))))
+            def json = slurper.parseText(it)
+            trusted.add(new Persona(new ByteArrayInputStream(Base64.decode(json.persona))))
         }
         def distrusted = new HashSet<>()
         persistBad.eachLine {
-            distrusted.add(new Persona(new ByteArrayInputStream(Base64.decode(it))))
+            def json = slurper.parseText(it)
+            distrusted.add(new Persona(new ByteArrayInputStream(Base64.decode(json.persona))))
         }
 
         assert trusted.size() == 1

doc/architecture.md

@@ -0,0 +1,59 @@
+# MuWire architecture
+
+### Core-UI separation
+
+The MuWire application is split conceptually into a `core` component and two `ui` components - one graphical component which is build using Swing and one text-only component built using the "lanterna" library.
+
+The core is written in mixture of Java and Groovy and is designed to be easy to embed into any application or language running on a JVM.  To achieve this, all communicatioon between the core and the outside world happens over an event bus using event objects.
+
+### Event bus and events
+
+At the heart of the core is the event bus.  It allows the different components that comprise the core to be decoupled, and allows the external components like UIs to communicate in asynchronous fashion with the core.
+
+The Core object has a single instance of the `com.muwire.core.EventBus` class.  It is responsible for dispatching events to any registered listeners.  Events themselves extend the `com.muwire.core.Event` class and carry arbitrary information relevant to the event.  See below or an example how to build a custom event:
+
+1.  Define the event in a class that extends `com.muwire.core.Event`:
+```
+package mypackage
+import com.muwire.core.Event
+
+class MyEvent extends Event {
+    // add relevant fields here
+}
+```
+
+2. Define one or more classes that will be notified of your events:
+```
+package mypackage
+
+class MyEventListener {
+    // ... add other logic here
+
+    void onMyEvent(MyEvent e) {
+        // logic to handle your type of event
+    }
+}
+```
+
+3. Register your event listener with the event bus:
+```
+MyEventListener myListener = new MyEventListener()
+eventBus.register(MyEvent.class,myListener)
+```
+You can register more than one listener for the same type of event; they will be notified in the order you register them.
+
+4. Publish events to the event bus
+```
+MyEvent myEvent = new MyEvent()
+// ... set relevant fields of the event ...
+eventBus.publish(myEvent)
+```
+
+Threading: the event bus creates a dedicated thread and all events are dispatched on that thread, regardless which thread publishes them.
+
+### Sharing files
+The UI publishes an event of type `com.muwire.core.files.FileSharedEvent` which contains a `java.io.File` reference to the file the user has chosen to share.  A component in the core called `HasherService` listens for these events, and when it receives notification that a FileSharedEvent has been posted it pereforms some sanity checks, then offloads the actual hashing to a dedicated thread.
+
+Before the hashing begins, another event of type `com.muwire.core.files.FileHashingEvent` is published that contains the name of the file.  At the moment that event serves only to update the UI with the current file being hashed.
+
+When the hashing completes, a `com.muwire.core.files.FileHashedEvent` is published by the HasherService.  The UI listens to this event and updates its list of shared files.  Another core component called `FileManager` also listens for such events and updates the interenal search index from the file name.