Merge remote-tracking branch 'remotes/origin/develop' into feature/object-hashtags-rework

# Conflicts:
#	CHANGELOG.md
#	lib/pleroma/web/activity_pub/activity_pub.ex
This commit is contained in:
Ivan Tashkinov 2021-02-03 09:31:38 +03:00
commit 4e14945670
130 changed files with 809 additions and 61 deletions

View File

@ -21,6 +21,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Admin API: Reports now ordered by newest
- Deprecated `Pleroma.Uploaders.S3, :public_endpoint`. Now `Pleroma.Upload, :base_url` is the standard configuration key for all uploaders.
- Improved Apache webserver support: updated sample configuration, MediaProxy cache invalidation verified with the included sample script
- Improve OAuth 2.0 provider support. A missing `fqn` field was added to the response, but does not expose the user's email address.
- Extracted object hashtags into separate table in order to improve hashtag timeline performance (via background migration in `Pleroma.Migrators.HashtagsTableMigrator`).
### Added
@ -49,7 +50,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Mastodon API: User and conversation mutes can now auto-expire if `expires_in` parameter was given while adding the mute.
- Admin API: An endpoint to manage frontends.
- Streaming API: Add follow relationships updates.
- WebPush: Introduce `pleroma:chat_mention` and `pleroma:emoji_reaction` notification types
- WebPush: Introduce `pleroma:chat_mention` and `pleroma:emoji_reaction` notification types.
- Mastodon API: Add monthly active users to `/api/v1/instance` (`pleroma.stats.mau`).
- Mastodon API: Home, public, hashtag & list timelines accept `only_media`, `remote` & `local` parameters for filtration.
- Mastodon API: `/api/v1/accounts/:id` & `/api/v1/mutes` endpoints accept `with_relationships` parameter and return filled `pleroma.relationship` field.
</details>
### Fixed
@ -59,6 +63,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Creating incorrect IPv4 address-style HTTP links when encountering certain numbers.
- Reblog API Endpoint: Do not set visibility parameter to public by default and let CommonAPI to infer it from status, so a user can reblog their private status without explicitly setting reblog visibility to private.
- Tag URLs in statuses are now absolute
- Removed duplicate jobs to purge expired activities
- File extensions of some attachments were incorrectly changed. This feature has been disabled for now.
- Mix task pleroma.instance creates missing parent directories if the configuration or SQL output paths are changed.
<details>
<summary>API Changes</summary>

View File

@ -0,0 +1 @@
See `Authentication` section of [the configuration cheatsheet](../configuration/cheatsheet.md#authentication).

View File

@ -893,6 +893,22 @@ Pleroma account will be created with the same name as the LDAP user name.
Note, if your LDAP server is an Active Directory server the correct value is commonly `uid: "cn"`, but if you use an
OpenLDAP server the value may be `uid: "uid"`.
### :oauth2 (Pleroma as OAuth 2.0 provider settings)
OAuth 2.0 provider settings:
* `token_expires_in` - The lifetime in seconds of the access token.
* `issue_new_refresh_token` - Keeps old refresh token or generate new refresh token when to obtain an access token.
* `clean_expired_tokens` - Enable a background job to clean expired oauth tokens. Defaults to `false`.
OAuth 2.0 provider and related endpoints:
* `POST /api/v1/apps` creates client app basing on provided params.
* `GET/POST /oauth/authorize` renders/submits authorization form.
* `POST /oauth/token` creates/renews OAuth token.
* `POST /oauth/revoke` revokes provided OAuth token.
* `GET /api/v1/accounts/verify_credentials` (with proper `Authorization` header or `access_token` URI param) returns user info on requester (with `acct` field containing local nickname and `fqn` field containing fully-qualified nickname which could generally be used as email stub for OAuth software that demands email field in identity endpoint response, like Peertube).
### OAuth consumer mode
OAuth consumer mode allows sign in / sign up via external OAuth providers (e.g. Twitter, Facebook, Google, Microsoft, etc.).
@ -965,14 +981,6 @@ config :ueberauth, Ueberauth,
]
```
### OAuth 2.0 provider - :oauth2
Configure OAuth 2 provider capabilities:
* `token_expires_in` - The lifetime in seconds of the access token.
* `issue_new_refresh_token` - Keeps old refresh token or generate new refresh token when to obtain an access token.
* `clean_expired_tokens` - Enable a background job to clean expired oauth tokens. Defaults to `false`.
## Link parsing
### :uri_schemes

View File

@ -16,6 +16,12 @@ Adding the parameter `reply_visibility` to the public and home timelines queries
Adding the parameter `instance=lain.com` to the public timeline will show only statuses originating from `lain.com` (or any remote instance).
Home, public, hashtag & list timelines accept these parameters:
- `only_media`: show only statuses with media attached
- `local`: show only local statuses
- `remote`: show only remote statuses
## Statuses
- `visibility`: has additional possible values `list` and `local` (for local-only statuses)
@ -54,6 +60,23 @@ The `id` parameter can also be the `nickname` of the user. This only works in th
- `/api/v1/accounts/:id`
- `/api/v1/accounts/:id/statuses`
`/api/v1/accounts/:id/statuses` endpoint accepts these parameters:
- `pinned`: include only pinned statuses
- `tagged`: with tag
- `only_media`: include only statuses with media attached
- `with_muted`: include statuses/reactions from muted accounts
- `exclude_reblogs`: exclude reblogs
- `exclude_replies`: exclude replies
- `exclude_visibilities`: exclude visibilities
Endpoints which accept `with_relationships` parameter:
- `/api/v1/accounts/:id`
- `/api/v1/accounts/:id/followers`
- `/api/v1/accounts/:id/following`
- `/api/v1/mutes`
Has these additional fields under the `pleroma` object:
- `ap_id`: nullable URL string, ActivityPub id of the user

View File

@ -242,6 +242,13 @@ def run(["gen" | rest]) do
rum_enabled: rum_enabled
)
config_dir = Path.dirname(config_path)
psql_dir = Path.dirname(psql_path)
[config_dir, psql_dir, static_dir, uploads_dir]
|> Enum.reject(&File.exists?/1)
|> Enum.map(&File.mkdir_p!/1)
shell_info("Writing config to #{config_path}.")
File.write(config_path, result_config)
@ -275,10 +282,6 @@ defp write_robots_txt(static_dir, indexable, template_dir) do
indexable: indexable
)
unless File.exists?(static_dir) do
File.mkdir_p!(static_dir)
end
robots_txt_path = Path.join(static_dir, "robots.txt")
if File.exists?(robots_txt_path) do

View File

@ -146,6 +146,7 @@ defmodule Pleroma.User do
field(:inbox, :string)
field(:shared_inbox, :string)
field(:accepts_chat_messages, :boolean, default: nil)
field(:last_active_at, :naive_datetime)
embeds_one(
:notification_settings,
@ -2030,6 +2031,15 @@ def local_nickname(nickname_or_mention) do
|> hd()
end
def full_nickname(%User{} = user) do
if String.contains?(user.nickname, "@") do
user.nickname
else
%{host: host} = URI.parse(user.ap_id)
user.nickname <> "@" <> host
end
end
def full_nickname(nickname_or_mention),
do: String.trim_leading(nickname_or_mention, "@")
@ -2444,4 +2454,19 @@ def sanitize_html(%User{} = user, filter) do
def get_host(%User{ap_id: ap_id} = _user) do
URI.parse(ap_id).host
end
def update_last_active_at(%__MODULE__{local: true} = user) do
user
|> cast(%{last_active_at: NaiveDateTime.utc_now()}, [:last_active_at])
|> update_and_set_cache()
end
def active_user_count(weeks \\ 4) do
active_after = Timex.shift(NaiveDateTime.utc_now(), weeks: -weeks)
__MODULE__
|> where([u], u.last_active_at >= ^active_after)
|> where([u], u.local == true)
|> Repo.aggregate(:count)
end
end

View File

@ -824,6 +824,12 @@ defp restrict_local(query, %{local_only: true}) do
defp restrict_local(query, _), do: query
defp restrict_remote(query, %{remote: true}) do
from(activity in query, where: activity.local == false)
end
defp restrict_remote(query, _), do: query
defp restrict_actor(query, %{actor_id: actor_id}) do
from(activity in query, where: activity.actor == ^actor_id)
end
@ -1198,6 +1204,7 @@ def fetch_activities_query(recipients, opts \\ %{}) do
|> restrict_replies(opts)
|> restrict_since(opts)
|> restrict_local(opts)
|> restrict_remote(opts)
|> restrict_actor(opts)
|> restrict_type(opts)
|> restrict_state(opts)

View File

@ -99,7 +99,10 @@ def show_operation do
summary: "Account",
operationId: "AccountController.show",
description: "View information about a profile.",
parameters: [%Reference{"$ref": "#/components/parameters/accountIdOrNickname"}],
parameters: [
%Reference{"$ref": "#/components/parameters/accountIdOrNickname"},
with_relationships_param()
],
responses: %{
200 => Operation.response("Account", "application/json", Account),
401 => Operation.response("Error", "application/json", ApiError),
@ -130,7 +133,7 @@ def statuses_operation do
:with_muted,
:query,
BooleanLike,
"Include statuses from muted acccounts."
"Include statuses from muted accounts."
),
Operation.parameter(:exclude_reblogs, :query, BooleanLike, "Exclude reblogs"),
Operation.parameter(:exclude_replies, :query, BooleanLike, "Exclude replies"),
@ -144,7 +147,7 @@ def statuses_operation do
:with_muted,
:query,
BooleanLike,
"Include reactions from muted acccounts."
"Include reactions from muted accounts."
)
] ++ pagination_params(),
responses: %{
@ -347,7 +350,7 @@ def mutes_operation do
operationId: "AccountController.mutes",
description: "Accounts the user has muted.",
security: [%{"oAuth" => ["follow", "read:mutes"]}],
parameters: pagination_params(),
parameters: [with_relationships_param() | pagination_params()],
responses: %{
200 => Operation.response("Accounts", "application/json", array_of_accounts())
}

View File

@ -25,6 +25,8 @@ def home_operation do
security: [%{"oAuth" => ["read:statuses"]}],
parameters: [
local_param(),
remote_param(),
only_media_param(),
with_muted_param(),
exclude_visibilities_param(),
reply_visibility_param() | pagination_params()
@ -61,6 +63,7 @@ def public_operation do
local_param(),
instance_param(),
only_media_param(),
remote_param(),
with_muted_param(),
exclude_visibilities_param(),
reply_visibility_param() | pagination_params()
@ -107,6 +110,7 @@ def hashtag_operation do
),
local_param(),
only_media_param(),
remote_param(),
with_muted_param(),
exclude_visibilities_param() | pagination_params()
],
@ -132,6 +136,9 @@ def list_operation do
required: true
),
with_muted_param(),
local_param(),
remote_param(),
only_media_param(),
exclude_visibilities_param() | pagination_params()
],
operationId: "TimelineController.list",
@ -198,4 +205,13 @@ defp only_media_param do
"Show only statuses with media attached?"
)
end
defp remote_param do
Operation.parameter(
:remote,
:query,
%Schema{allOf: [BooleanLike], default: false},
"Show only remote statuses?"
)
end
end

View File

@ -269,10 +269,14 @@ def relationships(%{assigns: %{user: user}} = conn, %{id: id}) do
def relationships(%{assigns: %{user: _user}} = conn, _), do: json(conn, [])
@doc "GET /api/v1/accounts/:id"
def show(%{assigns: %{user: for_user}} = conn, %{id: nickname_or_id}) do
def show(%{assigns: %{user: for_user}} = conn, %{id: nickname_or_id} = params) do
with %User{} = user <- User.get_cached_by_nickname_or_id(nickname_or_id, for: for_user),
:visible <- User.visible_for(user, for_user) do
render(conn, "show.json", user: user, for: for_user)
render(conn, "show.json",
user: user,
for: for_user,
embed_relationships: embed_relationships?(params)
)
else
error -> user_visibility_error(conn, error)
end
@ -454,7 +458,12 @@ def mutes(%{assigns: %{user: user}} = conn, params) do
conn
|> add_link_headers(users)
|> render("index.json", users: users, for: user, as: :user)
|> render("index.json",
users: users,
for: user,
as: :user,
embed_relationships: embed_relationships?(params)
)
end
@doc "GET /api/v1/blocks"

View File

@ -51,6 +51,8 @@ def home(%{assigns: %{user: user}} = conn, params) do
|> Map.put(:reply_filtering_user, user)
|> Map.put(:announce_filtering_user, user)
|> Map.put(:user, user)
|> Map.put(:local_only, params[:local])
|> Map.delete(:local)
activities =
[user.ap_id | User.following(user)]
@ -190,6 +192,7 @@ def list(%{assigns: %{user: user}} = conn, %{list_id: id} = params) do
|> Map.put(:blocking_user, user)
|> Map.put(:user, user)
|> Map.put(:muting_user, user)
|> Map.put(:local_only, params[:local])
# we must filter the following list for the user to avoid leaking statuses the user
# does not actually have permission to see (for more info, peruse security issue #270).

View File

@ -262,7 +262,9 @@ defp do_render("show.json", %{user: user} = opts) do
}
},
# Pleroma extension
# Pleroma extensions
# Note: it's insecure to output :email but fully-qualified nickname may serve as safe stub
fqn: User.full_nickname(user),
pleroma: %{
ap_id: user.ap_id,
also_known_as: user.also_known_as,

View File

@ -45,6 +45,7 @@ def render("show.json", _) do
fields_limits: fields_limits(),
post_formats: Config.get([:instance, :allowed_post_formats])
},
stats: %{mau: Pleroma.User.active_user_count()},
vapid_public_key: Keyword.get(Pleroma.Web.Push.vapid_config(), :public_key)
}
}

View File

@ -0,0 +1,30 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.Plugs.UserTrackingPlug do
alias Pleroma.User
import Plug.Conn, only: [assign: 3]
@update_interval :timer.hours(24)
def init(opts), do: opts
def call(%{assigns: %{user: %User{id: id} = user}} = conn, _) when not is_nil(id) do
with true <- needs_update?(user),
{:ok, user} <- User.update_last_active_at(user) do
assign(conn, :user, user)
else
_ -> conn
end
end
def call(conn, _), do: conn
defp needs_update?(%User{last_active_at: nil}), do: true
defp needs_update?(%User{last_active_at: last_active_at}) do
NaiveDateTime.diff(NaiveDateTime.utc_now(), last_active_at, :millisecond) >= @update_interval
end
end

View File

@ -56,6 +56,7 @@ defmodule Pleroma.Web.Router do
plug(Pleroma.Web.Plugs.UserEnabledPlug)
plug(Pleroma.Web.Plugs.SetUserSessionIdPlug)
plug(Pleroma.Web.Plugs.EnsureUserTokenAssignsPlug)
plug(Pleroma.Web.Plugs.UserTrackingPlug)
end
pipeline :base_api do
@ -319,6 +320,8 @@ defmodule Pleroma.Web.Router do
end
scope "/oauth", Pleroma.Web.OAuth do
# Note: use /api/v1/accounts/verify_credentials for userinfo of signed-in user
get("/registration_details", OAuthController, :registration_details)
post("/mfa/verify", MFAController, :verify, as: :mfa_verify)

View File

@ -6,7 +6,7 @@
</div>
<span class="display-name" style="padding-left: 0.5em;">
<bdi><%= raw (@author.name |> Formatter.emojify(@author.emoji)) %></bdi>
<span class="nickname"><%= full_nickname(@author) %></span>
<span class="nickname">@<%= full_nickname(@author) %></span>
</span>
</a>
</div>

View File

@ -17,6 +17,8 @@ defmodule Pleroma.Web.EmbedView do
use Phoenix.HTML
defdelegate full_nickname(user), to: User
@media_types ["image", "audio", "video"]
defp fetch_media_type(%{"mediaType" => mediaType}) do
@ -30,11 +32,6 @@ defp open_content? do
)
end
defp full_nickname(user) do
%{host: host} = URI.parse(user.ap_id)
"@" <> user.nickname <> "@" <> host
end
defp status_title(%Activity{object: %Object{data: %{"name" => name}}}) when is_binary(name),
do: name

View File

@ -17,12 +17,14 @@ def perform(%Job{
"object" => %{"data" => %{"attachment" => [_ | _] = attachments, "actor" => actor}}
}
}) do
attachments
|> Enum.flat_map(fn item -> Enum.map(item["url"], & &1["href"]) end)
|> fetch_objects
|> prepare_objects(actor, Enum.map(attachments, & &1["name"]))
|> filter_objects
|> do_clean
if Pleroma.Config.get([:instance, :cleanup_attachments], false) do
attachments
|> Enum.flat_map(fn item -> Enum.map(item["url"], & &1["href"]) end)
|> fetch_objects
|> prepare_objects(actor, Enum.map(attachments, & &1["name"]))
|> filter_objects
|> do_clean
end
{:ok, :success}
end

View File

@ -7,7 +7,7 @@ defmodule Pleroma.Workers.PurgeExpiredActivity do
Worker which purges expired activity.
"""
use Oban.Worker, queue: :activity_expiration, max_attempts: 1
use Oban.Worker, queue: :activity_expiration, max_attempts: 1, unique: [period: :infinity]
import Ecto.Query

View File

@ -194,7 +194,7 @@ defp deps do
{:restarter, path: "./restarter"},
{:majic,
git: "https://git.pleroma.social/pleroma/elixir-libraries/majic.git",
ref: "4c692e544b28d1f5e543fb8a44be090f8cd96f80"},
ref: "289cda1b6d0d70ccb2ba508a2b0bd24638db2880"},
{:open_api_spex,
git: "https://git.pleroma.social/pleroma/elixir-libraries/open_api_spex.git",
ref: "f296ac0924ba3cf79c7a588c4c252889df4c2edd"},

View File

@ -66,7 +66,7 @@
"jumper": {:hex, :jumper, "1.0.1", "3c00542ef1a83532b72269fab9f0f0c82bf23a35e27d278bfd9ed0865cecabff", [:mix], [], "hexpm", "318c59078ac220e966d27af3646026db9b5a5e6703cb2aa3e26bcfaba65b7433"},
"libring": {:hex, :libring, "1.4.0", "41246ba2f3fbc76b3971f6bce83119dfec1eee17e977a48d8a9cfaaf58c2a8d6", [:mix], [], "hexpm"},
"linkify": {:hex, :linkify, "0.4.1", "f881eb3429ae88010cf736e6fb3eed406c187bcdd544902ec937496636b7c7b3", [:mix], [], "hexpm", "ce98693f54ae9ace59f2f7a8aed3de2ef311381a8ce7794804bd75484c371dda"},
"majic": {:git, "https://git.pleroma.social/pleroma/elixir-libraries/majic.git", "4c692e544b28d1f5e543fb8a44be090f8cd96f80", [ref: "4c692e544b28d1f5e543fb8a44be090f8cd96f80"]},
"majic": {:git, "https://git.pleroma.social/pleroma/elixir-libraries/majic.git", "289cda1b6d0d70ccb2ba508a2b0bd24638db2880", [ref: "289cda1b6d0d70ccb2ba508a2b0bd24638db2880"]},
"makeup": {:hex, :makeup, "1.0.3", "e339e2f766d12e7260e6672dd4047405963c5ec99661abdc432e6ec67d29ef95", [:mix], [{:nimble_parsec, "~> 0.5", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "2e9b4996d11832947731f7608fed7ad2f9443011b3b479ae288011265cdd3dad"},
"makeup_elixir": {:hex, :makeup_elixir, "0.14.1", "4f0e96847c63c17841d42c08107405a005a2680eb9c7ccadfd757bd31dabccfb", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm", "f2438b1a80eaec9ede832b5c41cd4f373b38fd7aa33e3b22d9db79e640cbde11"},
"meck": {:hex, :meck, "0.8.13", "ffedb39f99b0b99703b8601c6f17c7f76313ee12de6b646e671e3188401f7866", [:rebar3], [], "hexpm", "d34f013c156db51ad57cc556891b9720e6a1c1df5fe2e15af999c84d6cebeb1a"},

View File

@ -0,0 +1,11 @@
defmodule Pleroma.Repo.Migrations.AddLastActiveAtToUsers do
use Ecto.Migration
def change do
alter table(:users) do
add(:last_active_at, :naive_datetime)
end
create_if_not_exists(index(:users, [:last_active_at]))
end
end

View File

@ -0,0 +1,29 @@
defmodule Pleroma.Repo.Migrations.RemoveDuplicatesFromActivityExpirationQueue do
use Ecto.Migration
import Ecto.Query, only: [from: 2]
def up do
duplicate_ids =
from(j in Oban.Job,
where: j.queue == "activity_expiration",
where: j.worker == "Pleroma.Workers.PurgeExpiredActivity",
where: j.state == "scheduled",
select:
{fragment("(?)->>'activity_id'", j.args), fragment("array_agg(?)", j.id), count(j.id)},
group_by: fragment("(?)->>'activity_id'", j.args),
having: count(j.id) > 1
)
|> Pleroma.Repo.all()
|> Enum.map(fn {_, ids, _} ->
max_id = Enum.max(ids)
List.delete(ids, max_id)
end)
|> List.flatten()
from(j in Oban.Job, where: j.id in ^duplicate_ids)
|> Pleroma.Repo.delete_all()
end
def down, do: :noop
end

View File

@ -1 +1 @@
<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1,user-scalable=no"><!--server-generated-meta--><link rel=icon type=image/png href=/favicon.png><link href=/static/css/app.9a4c5ede37b2f0230836.css rel=stylesheet></head><body class=hidden><noscript>To use Pleroma, please enable JavaScript.</noscript><div id=app></div><script type=text/javascript src=/static/js/vendors~app.54838a79dee084ec3dad.js></script><script type=text/javascript src=/static/js/app.eb8f7164fc75862a251d.js></script></body></html>
<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1,user-scalable=no"><!--server-generated-meta--><link rel=icon type=image/png href=/favicon.png><link href=/static/css/app.9a4c5ede37b2f0230836.css rel=stylesheet></head><body class=hidden><noscript>To use Pleroma, please enable JavaScript.</noscript><div id=app></div><script type=text/javascript src=/static/js/vendors~app.3b02e2e5bd8cdca42216.js></script><script type=text/javascript src=/static/js/app.c6b8a1c86149ed63e6ff.js></script></body></html>

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More